SMSTC (2007/08)

Probability

                www.smstc.ac.uk

Contents

3 Random variables 3–1 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–2 3.2 Random variables are functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–2 3.3 Continuous functions of random variables . . . . . . . . . . . . . . . . . . . . . . 3–3 3.4 Induced σ-fields; measurability . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–4 3.5 Law of a random variable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–7 3.6 Law of a discrete random variable . . . . . . . . . . . . . . . . . . . . . . . . . . 3–8 3.7 Tossing coins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–9 3.8 Uniform random variable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–13 3.9 Probability, revisited . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–15 3.10 Distribution functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–15 3.11 Types of distribution functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–17

3.11.1 Discrete distribution functions . . . . . . . . . . . . . . . . . . . . . . . . 3–17 3.11.2 Continuous distribution functions . . . . . . . . . . . . . . . . . . . . . . . 3–18 3.11.3 General distribution functions . . . . . . . . . . . . . . . . . . . . . . . . . 3–19 3.11.4 Differentiation: a word of caution . . . . . . . . . . . . . . . . . . . . . . . 3–19

3.12 Transformation rules and densities . . . . . . . . . . . . . . . . . . . . . . . . . . 3–20 3.13 Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–21

3.13.1 Definition of expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–22 3.13.2 Substitution rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–26 3.13.3 Expectation and densities . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–27

3.14 Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–28 3.14.1 Markov, Chebyshev, Chernoff . . . . . . . . . . . . . . . . . . . . . . . . . 3–29 3.14.2 Jensen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–29

3.15 Moments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–31 3.16 Hölder, Minkowski and Cauchy-Bunyakowskii-Schwarz . . . . . . . . . . . . . . . 3–31 3.17 Moment generating functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–32 3.18 APPENDIX A: Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–34 3.19 APPENDIX B: Sets and numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–37

0–1 SMST C: Probability (i)

4 Joint distributions and independence 4–1 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–1 4.2 Joint distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–1

4.2.1 Knowledge of F_{X,Y} implies knowledge of P_{X,Y} . . . . . . . . . . . . . . . 4–2 4.3 Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–3 4.4 Joint density . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–4 4.5 Joint moment generating function . . . . . . . . . . . . . . . . . . . . . . . . . . 4–5 4.6 Correlations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–5 4.7 Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–7

4.7.1 Naïve conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–7 4.7.2 Geometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–7 4.7.3 Conditional probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–9 4.7.4 Conditional law . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–10

4.8 Gaussian variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–12 4.8.1 The Gaussian law . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–12 4.8.2 The multidimensional Gaussian random variable . . . . . . . . . . . . . . 4–13 4.8.3 Conditional Gaussian law . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–16

5 Important special distributions 5–1 5.1 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5–1 5.2 Binomial, Poisson, and multinomial . . . . . . . . . . . . . . . . . . . . . . . . . . 5–1 5.3 Thinning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5–3 5.4 Geometric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5–4 5.5 Uniform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5–5 5.6 Exponential . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5–6

Probability

3: Random variables

Takis Konstantopoulos^{a}

             www.smstc.ac.uk

\underline{Contents}

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–2 3.2 Random variables are functions . . . . . . . . . . . . . . . . . . . . . . 3–2 3.3 Continuous functions of random variables . . . . . . . . . . . . . . . . 3–3 3.4 Induced σ-fields; measurability . . . . . . . . . . . . . . . . . . . . . . 3–4 3.5 Law of a random variable . . . . . . . . . . . . . . . . . . . . . . . . . 3–7 3.6 Law of a discrete random variable . . . . . . . . . . . . . . . . . . . . 3–8 3.7 Tossing coins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–9 3.8 Uniform random variable . . . . . . . . . . . . . . . . . . . . . . . . . . 3–13 3.9 Probability, revisited . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–15 3.10 Distribution functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–15 3.11 Types of distribution functions . . . . . . . . . . . . . . . . . . . . . . 3–17

3.11.1 Discrete distribution functions . . . . . . . . . . . . . . . . . . . . . . . 3–17 3.11.2 Continuous distribution functions . . . . . . . . . . . . . . . . . . . . . . 3–18 3.11.3 General distribution functions . . . . . . . . . . . . . . . . . . . . . . . . 3–19 3.11.4 Differentiation: a word of caution . . . . . . . . . . . . . . . . . . . . . . 3–19

3.12 Transformation rules and densities . . . . . . . . . . . . . . . . . . . . 3–20 3.13 Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–21

3.13.1 Definition of expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–22 3.13.2 Substitution rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–26 3.13.3 Expectation and densities . . . . . . . . . . . . . . . . . . . . . . . . . . 3–27

3.14 Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–28 3.14.1 Markov, Chebyshev, Chernoff . . . . . . . . . . . . . . . . . . . . . . . . 3–29 3.14.2 Jensen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–29

3.15 Moments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–31 3.16 Hölder, Minkowski and Cauchy-Bunyakowskii-Schwarz . . . . . . . . 3–31 3.17 Moment generating functions . . . . . . . . . . . . . . . . . . . . . . . 3–32 3.18 APPENDIX A: Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–34 3.19 APPENDIX B: Sets and numbers . . . . . . . . . . . . . . . . . . . . 3–37

3–1

SMST C: Probability 3–2

3.1 Introduction

Random variables are at the core of Probability and Modern Mathematics and have nothing to do with randomness: they are merely functions, labels of outcomes of an “abstract space” Ω. These functions must be compatible with the set of events of Ω. This compatibility property is called measurability. The concept of a random variable is purely functional.

David Mumford [4] challenges the mathematical community by insisting that we are encountering a paradigm shift in Mathematics, namely, we are seeing how “Stochasticity” plays a much more fundamental role within Mathematics than previously envisioned. Moreover, Mumford claims that random variables should be put first in the discipline of Probability. Somehow, he suggests a change of the neo-classical Kolmogorov foundation, and wants to replace it by functions, i.e. random variables, as primary objects. There is no doubt that this can be done, just as Topology can be approached, not with sets, but with functions to start with. While we will not be as “radical” as Mumford, we will explain the importance of random variables by explicitly discussing all (classical) nuances associated with them.

So we first start with functional concepts, ignoring probability. We then put a probability and see how it is transformed by a random variable: a random variable transforms a probability into another probability that is known as its law.

Discrete random variables are trivial, from a foundational point of view (but far from trivial otherwise). Continuous random variables are hard to visualise. We prefer to construct such random variables. by flipping coins which is the most natural thing to do in Probability (arguably the only thing to do).

The distribution of a random variable is any function that uniquely specifies its law. It is a convention, for real random variables, to use the so-called distribution function as a distribution. Some distribution functions are nice in the sense that they can be differentiated and their derivatives are useful because they can be integrated to enable us to compute probabilities. These derivatives are called densities. We try to be semi-honest when discussing them.

The expectation of a random variable is discussed, constructed, and shown to be compatible with the usual naïve formulae.

Some sine qua non inequalities (under the names of Markov, Chebyshev, Chernoff, Jensen, Cauchy, Bunyakowskii, Schwarz, Hölder and Minkowski) are presented, because inequalities are more useful than equalities.

Finally, there is a brief discussion of he concept of a moment generating function.

Exercises are scattered within the text.

Appendix A summarises basic notions regarding sets. Appendix B is gives a somewhat elementary approach to putting probabilities on sets.

3.2 Random variables are functions

A measurable space is a set together with a σ-field of subsets of it. The concept of a random variable was introduced in 1.1.5 . Intuitively, a random variable assigns concrete labels to abstract outcomes. Concretely,

A random variable is a measurable function X between two measurable spaces, an “abstract” one, (Ω, ℱ), and a “concrete”^{a} one, (S, 𝒮).

aThe adjectives in quotes have nothing to do with Mathematics but, rather, with our human interpretation of it.

In other words, we require that the inverse image by X of each element of 𝒮 be an element of ℱ. We denote this situation by

X : (Ω, ℱ) → (S, 𝒮).

Many a times (Ω, ℱ) could (or should) be left unspecified but (S, 𝒮) should be chosen specifically; for example it could be (ℝ, ℬ), where ℬ = ℬ(ℝ) is the class of Borel sets (see 1.1.3 ) on ℝ. In such a case we call X ONE (real) random variable, omitting the numeral ONE when not necessary. As another example, let (S, 𝒮) = (ℝ^{2}, ℬ(ℝ^{2})), where ℬ(ℝ^{2}) are the Borel sets on ℝ^{2} (defined as the smallest σ-field containing all open sets in ℝ^{2}). In this case we refer to X as TWO (real) random variables, because we may, by choosing Cartesian coördinates on ℝ^{2}, represent X by (X_{1}, X_{2}), where X_{1} is one random variable and X_{2} is also one random variable. More generally, we may let (S, 𝒮) = (ℝ^{d}, ℬ(ℝ^{d})), and then we refer to X a random (d-dimensional real) vector.

Terminology: we say that X is a random variables IN S or that X is a random element OF S. If S is a concrete space with a name “abc” then we call X a random “abc”. For instance, if S is the space of triangles on the plane, then a random variable in X is called random triangle.

EXERCISE 1. Show that if X : (Ω, ℱ) → (S, 𝒮), H : (S, 𝒮) → (T, 𝒯 ) are random variables then H^{∘}X is a random variable.

The following lemma, useful in checking that a function is a random variable:

Lemma 3.1. If X : Ω → S is a function, ℱ a σ-field on Ω and 𝒮 a σ-field on S generated by the collection of sets 𝒞 then X is a random variable if and only if X^{−1}(B) ∈ ℱ for all B ∈ 𝒞.

Proof Let 𝒮′ := {B ⊂ S : X^{−1}(B) ∈ ℱ }. This is a σ-field. Indeed, if B ∈ 𝒮′ then X^{−1}(B^{c}) = X^{−1}(B)^{c} ∈ ℱ because X^{−1}(B) ∈ ℱ. If B_{1}, B_{2}, … ∈ 𝒮′ then X^{−1}(∩_{j}B_{j}) = ∩_{j}X^{−1}(B_{j}) ∈ ℱ because each X^{−1}(B_{j}) ∈ ℱ. Suppose that X^{−1}(B) ∈ ℱ for all B ∈ 𝒞. This means that 𝒞 is contained in 𝒮′. But then 𝒮 which is the smallest σ-field containing 𝒞 must be contained in 𝒮′ for the latter is a σ-field. In symbols, 𝒮 ⊂ 𝒮′. But, by definition of 𝒮′, this means that X^{−1}(B) ∈ ℱ for any B ∈ 𝒮. □

Recall that the notation

{X ∈ B} = X^{−1}(B)

is used all the time, so we will stick to it when we can.

Corollary 3.1. If X : Ω → ℝ is a function and ℱ a σ-field on Ω, then X is one random variable ⟺ {X ≤ x} ∈ ℱ for all x ∈ ℝ ⟺ {X < x} ∈ ℱ for all x ∈ ℝ ⟺ {X > x} ∈ ℱ for all x ∈ ℝ.

The real fun starts when (S, 𝒮) is chosen to be a large (but concrete) space, for example, S can be a subset of a space of real-valued functions on the real line and 𝒮 a suitable σ-field. In such a case, X is called random function or stochastic process (stochastic being a synonym of random and process being a synonym of function; therefore, random process or stochastic function are also acceptable terminologies for the same thing).

3.3 Continuous functions of random variables

Lemma 3.2. If X is a random vector in ℝ^{d} and f : ℝ^{d} → ℝ^{k} is a continuous function then f^{∘}X is a random vector in ℝ^{k}.

Proof Let B be an open subset of ℝ^{k}. Then f^{−1}(B) is an open subset of ℝ^{d} because f is continuous. Hence f^{−1}(B) is a Borel subset of ℝ^{d} and so f is measurable. Now use Exercise 1. □

An application of this is that all usual algebraic operations on random variables will result random variables. So

X_{1} + X_{2}, X_{1} · X_{2}, X_{1} ∧ X_{2}

are all random variables.

A more important result is that we can go beyond algebraic operations and retain measurability. So

Lemma 3.3. If X_{1}, X_{2}, … are random variables in ℝ then

inf_{j} X_{j}, \underline{lim}_{j} X_{j}, sup_{j} X_{j}, \overline{lim}_{j} X_{j}

are random variables in ℝ ∪ {+∞, −∞}.

Proof Indeed,

{inf_{j} X_{j} ≤ a} = ∩_{j}{X_{j} ≤ a}

and, since {X_{j} ≤ a} ∈ ℱ for all j, and ℱ is a σ-field, then the intersection also belongs to ℱ. For the second variable, we simply observe that sup_{j} X_{j} = − inf_{j}(−X_{j}) is also a random variable and so \underline{lim}_{j} X_{j} = sup_{j} inf_{k≥j} X_{k} is also a random variable. □

3.4 Induced σ-fields; measurability

Recall, as in 1.1.3 , that

the σ-field generated by a collection, say 𝒜 of subsets of Ω, is defined as the intersection of all σ-fields containing 𝒜; it is denoted by σ(𝒜).

If X : Ω → S is a function, and if 𝒮 is a fixed σ-field on S, the σ-field generated or induced by X is defined by

σ(X) := {X^{−1}(B), B ∈ 𝒮 }.

It is easy to see that

Lemma 3.4. (i) σ(X) is a σ-field, (ii) σ(X) is the intersection of all σ-fields 𝒢 ⊂ ℱ such that X : (Ω, 𝒢) → (S, 𝒮) is a random variable, (iii) X is a random variable if and only if σ(X) ⊂ ℱ.

If Y : Ω ↦ S′ is another function, and if S′ is endowed with another fixed σ-field 𝒮′, then the notation σ(X, Y) stands for the smallest σ-field containing σ(X) = {X^{−1}(B), B ∈ 𝒮 } and σ(Y) = {X^{−1}(B), B ∈ 𝒮′}:

σ(X, Y) := σ(σ(X) ∪ σ(Y)) =: σ(X) ∨ σ(Y).

(The last bit is just another notation for the same thing.) Let us study random variables that take values in ℝ^{d} or, more generally, in a product S_{1} × ⋯ × S_{d} of sets. Suppose that on each S_{i} we have a σ-field 𝒮_{i} ⊂ 2^{S_{i}}. We first construct a natural σ-field on S_{1} × ⋯ × S_{d}. For each i consider the projection function

π_{i} : S_{1} × ⋯ × S_{d} → S_{i}; π_{i} : (s_{1}, ⋯, s_{d}) ↦ s_{i}.

Define

𝒮_{1} ⊗ ⋯ ⊗ 𝒮_{d} := σ(π_{1}, …, π_{d}).

EXERCISE 2. Consider (S_{i}, 𝒮_{i}), i = 1, …, d. Let d = 2 for simplicity. Show that

𝒮_{1} ⊗ 𝒮_{2} = σ({B_{1} × S_{2} : B_{1} ∈ 𝒮_{1}} ∪ {S_{1} × B_{2} : B_{2} ∈ 𝒮_{2}}).
= σ({B_{1} × B_{2} : B_{1} ∈ 𝒮_{1}, B_{2} ∈ 𝒮_{2}}).

Lemma 3.5. Let X_{i} : (Ω, ℱ) → (S_{i}, 𝒮_{i}), i = 1, 2, …, d, be random variables. Let S = S_{1} × ⋯ × S_{d}, 𝒮 = 𝒮_{1} ⊗ ⋯ ⊗ 𝒮_{d}. Then (X_{1}, …, X_{d}) : (Ω, ℱ) → (S, 𝒮) is a random variable.

Proof We just have to check that (X_{1}, …, X_{d})^{−1}(B) ∈ ℱ for B ranging in a suitable π-system. We take this π-system (and it is immediate to see that it is so) to be all sets of the form B_{1} × ⋯ × B_{d}, where B_{j} ∈ 𝒮_{j} for all 1 ≤ j ≤ d. Then (X_{1}, …, X_{d})^{−1}(B_{1} × ⋯ × B_{d}) = {(X_{1}, …, X_{d}) ∈ B_{1} × ⋯ × B_{d}} = {X_{1} ∈ B_{1}} ∩ ⋯ ∩ {X_{d} ∈ B_{d}} ∈ ℱ. □

Moreover, and more importantly for all the theory of stochastic processes, we can carry this to infinity. Let (S_{i}, 𝒮_{i}), i = 1, 2, …, be measurable spaces. We wish to consider the space S = ×_{i=1}^{∞}S_{i} and endow it with a natural σ-field 𝒮. To do so, consider sets of the form B_{1} × B_{2} × ⋯^{b} where B_{i} ∈ 𝒮_{i} for all i, and where B_{j} = S_{j} for all but finitely many indices j. Define 𝒮 = ⨂_{i=1}^{∞} 𝒮_{i} to be the smallest σ field containing all these sets.

Lemma 3.6. Suppose that for each i ∈ ℕ, X_{i} : (Ω, ℱ) → (S_{i}, 𝒮_{i}) is a random variable. Let S = ×_{i=1}^{∞}S_{i} and let X = (X_{1}, X_{2}, …) : (Ω, ℱ) → (S, 𝒮), where 𝒮 = ⨂_{i=1}^{∞} 𝒮_{i} is the productsigma field. Then X is a random variable.

Proof Consider the π-system consisting of sets of the form ∩_{i=1}^{∞}B_{i} where B_{i} ∈ 𝒮_{i} for all i, and where B_{i} = S_{i} for all but finitely many indices i. □

Now consider the set ℝ^{2}. The natural σ-field on it if ℬ ⊗ ℬ, usually denoted by ℬ(ℝ^{2}). Similarly for ℝ^{d}.

We should view σ(X) as the essential information contained in X. We explain: If Ω is a set and X : Ω → S a function then just by looking at the values of X we may not be able to distinguish elements of Ω, simply because X may take the same value on different points of Ω. In other words, unless X is one-to-one, the set X(Ω) = {X(ω), ω ∈ Ω} contains “less information” than Ω. Take a trivial example: Let Ω = {−N, −N + 1, …, N − 1, N}, and let X(i) = i^{3} for all i ∈ Ω. Then X(Ω) contains precisely the same information as Ω. But if Y (i) = i^{2} then Y (Ω) is less informative than Ω because the sign is lost. In some sense, we are not interested in the exact values of a random variable but only on whether the values can distinguish elements of Ω.

Take an abstract set Ω (for example think of Ω as a square in the 2-dimensional Euclidean plane), let A ⊂ Ω and consider the random variable 1_{A}. This takes only two values, 0 and 1, so the information conveyed by it is not that large. Indeed,

σ(1_{A}) = {∅, A, A^{c}, Ω}

We may represent σ(1_{A}) as a partition of Ω, the partition consisting of A and its complement. Thus, 1_{A} can only tell us whether we are on A or on its complement: very little information indeed. But we may as well use any function of 1_{A} that does not compress it further, for example, c · 1_{A}, where c is a nonzero constant.

EXERCISE 3. Consider two subsets A_{1}, A_{2} of Ω and prove that the σ-field generated by c_{1}1_{A_{1}} + c_{2}1_{A_{2}} is

σ(c_{1}1_{A_{1}} + c_{2}1_{A_{2}}) = {∅, Ω, A_{1}, A_{2}, A_{1}^{c}, A_{2}^{c}, A_{1}A_{2}, A_{1}A_{2}^{c}, A_{1}^{c}A_{2}, A_{1}^{c}A_{2}^{c},

A_{1} ∪ A_{2}, A_{1} ∪ A_{2}^{c}, A_{1}^{c} ∪ A_{2}, A_{1}^{c} ∪ A_{2}^{c}, A_{1}△A_{2}, (A_{1}△A_{2})^{c}}.

bEach such set is called ‘finite-dimensional rectangle’.

It is more convenient to represent this σ-field by the partition of Ω induced by A_{1}, A_{2}, i.e. by the collection of disjoint sets

{A_{1}A_{2}, A_{1} \ A_{2}, A_{2} \ A_{1}, (A_{1}A_{2})^{c}}.

EXERCISE 4. Any nonempty set in Exercise 3 can be obtained as union of elements of this partition.

EXERCISE 5. Given a partition 𝒞 = {C_{1}, …, C_{n}} of Ω show that the σ-field generated by 𝒞 consists of the empty set and all sets that can be obtained by taking unions of sets in 𝒞. Assuming that none of the C_{i} is empty, this 𝒞 contains exactly 2^{n} sets. Also show that any random variable that takes value b_{i} on C_{i} for each i generates a σ-field which is contained in σ(𝒞). Show that if the values b_{i} are distinct then σ(X) = σ(𝒞).

Unfortunately, this partition point of view does not carry over to big spaces. It works well for finite Ω or for general Ω but small σ-fields, but fails in richer situations. However, the intuition obtained remains, in some sense, valid. Let us explore this further.

If X, Y are random variables on a common measurable space (Ω, ℱ) (and values in arbitrary sets) we say that Y is measurable with respect to X if

σ(Y) ⊂ σ(X).

We often write this as

Y ∈ σ(X).

For example, if X is one random variable then X^{2} is measurable with respect to X (but not vice-versa); if (X_{1}, X_{2}, X_{3}) are three random variables then (X_{1}, X_{2}) is measurable with respect to (X_{1}, X_{2}, X_{3}) and so is (cos(X_{1} + X_{2}), −X_{3} + log |X_{1}|). If it appears that

“measurability with respect to” means “(suitable) function of”

then this is because it really is so:

Lemma 3.7. If X : (Ω, ℱ) → (S, 𝒮), Y : (Ω, ℱ) → (ℝ, ℬ) are random variables, and if Y ∈ σ(X), then there exists a random variable (measurable map) H : (S, 𝒮) → (ℝ, ℬ) such that Y = H^{∘}X.

(The proof is deferred until the next page or so.) To understand why let us consider first a trivial situation: suppose that X takes only finitely many values; let x_{1}, …, x_{n} be these (distinct) values.

EXERCISE 6. Show that if X(Ω) = {x_{1}, …, x_{n}} is a finite set then σ(X) is generated by the partition {{X = x_{i}}, i = 1, …, n} of Ω.

EXERCISE 7. Using Exercise 5 show that any Y, measurable with respect to X, must be of the form

Y = ∑_{i=1}^{n} c_{i}1(X = x_{i}).

We then see that what is claimed in Lemma 3.7 is correct, in this special situation with X being finitely-valued. Indeed, let H(x) = ∑_{i=1}^{n} c_{i}1(x = x_{i}) and, obviously, Y = H^{∘}X.

The general case requires an approximation result that says that any measurable random variable X : (Ω, ℱ) → (ℝ, ℬ) can be approximated by simple random variables. A simple random variable is a random variable with finitely many values.

Lemma 3.8. Let X : (Ω, ℱ) → (ℝ, ℬ). Then there exists a sequence X_{1}, X_{2}, … of simple random variables such that lim_{n→∞} X_{n}(ω) = X(ω). If X(ω) ≥ 0, ω ∈ Ω, we can choose the sequence so that 0 ≤ X_{n}(ω) ≤ X_{n+1}(ω) for each n and ω.

Proof First suppose X(ω) ≥ 0 for all ω ∈ Ω. Let ⌈x⌉ denote the smallest integer k such that k ≥ x. Define

τ_{n}(x) := 2^{−n}⌈2^{n}x⌉ ∧ n.  (3.1)
X_{n}(ω) := τ_{n}(X(ω)).  (3.2)

That X_{n}(ω) → X(ω) as n → ∞ is obvious. Since ⌈a⌉ ≤ a we have 2⌈a⌉ ≤ ⌈2a⌉ and so 2⌈2^{n}X⌉ ≤ ⌈2^{n+1}X⌉. Dividing both sides with 2^{n+1} we have 2^{−n}⌈2^{n}X⌉ ≤ 2^{−n−1}⌈2^{n+1}X⌉ and so X_{n} ≤ X_{n+1}. Next do not place any requirement on the sign of X but observe that

X = X^{+} − X^{−}.

Then reduce to the previous case. □

Proof of Lemma 3.7 Suppose first that Y = 1_{A} for some A ∈ ℱ and that Y ∈ σ(X). This means that A ∈ σ(X). By definition of σ(X), there is a B ∈ ℬ such that A = X^{−1}(B). Define H(x) = 1(x ∈ B), x ∈ ℝ. Then H(X(ω)) = 1(X(ω) ∈ B) = 1(ω ∈ X^{−1}(B)) = 1(ω ∈ A) = Y (ω) and so H^{∘}X = Y, as required. Suppose next that Y is a simple random variable. Thus Y = ∑_{i=1}^{k} c_{i}1_{A_{i}} where A_{i} ∈ ℱ. By the previous case, there are H_{i} : (S, 𝒮) → (ℝ, ℬ), such that 1_{A_{i}} = H_{i}(X). Define H(x) = ∑_{i=1}^{k} c_{i}H_{i}(x). Then, clearly, H(X) = Y. Finally, suppose that Y : (Ω, ℱ) → (ℝ, ℬ) is a positive random variable. Then, by lemma 3.8, we can find simple Y_{n} : (Ω, ℱ) → (ℝ, ℬ), for all n ∈ ℕ, such that Y = sup_{n} Y_{n}. By the case analysed above, we have, for each n, a measurable function H_{n} such that Y_{n} = H_{n}^{∘}X. Define H = sup_{n} H_{n}. By Lemma 3.3, H is measurable. Also, H^{∘}X = (sup_{n} H_{n})^{∘}X = sup_{n}(H_{n}^{∘}X) = sup_{n} Y_{n} = Y, as required. In the general case, write Y = Y^{+} − Y^{−} and reduce to the above one. □

EXERCISE 8. Let X be one random variable in ℝ. Show that σ(X^{2}) is a strict subset of σ(X). But show that σ(2^{X}) = σ(X).

Remark Lemma 3.7 is very important in enhancing our understanding of measurability. Measurability, in Probability Theory (as well as in other areas of Mathematics like Descriptive Set Theory) is not an abstract notion but conveys precisely the idea of “information” contained in a measurement. It should also be stressed that Y in Lemma 3.7 does not have to be restricted to take values in ℝ. It could very well be a random vector (values in ℝ^{d}) or, more generally, a random element of a fairly arbitrary space known as Polish space. We shall not enter into such advanced details here.

3.5 Law of a random variable

The reader will of course have noticed that

the concept of “random variable” has nothing to do with randomness.

The adjective “random” is attached because, in addition to the above, we also consider some probability (measure) P on (Ω, ℱ) . As a mathematical definition, the concept of random variable is purely functional. So let us do that: let P be a probability on (Ω, ℱ). The distribution or law of the random variable X is a probability P_{X} on (S, 𝒮) which is induced, in the most natural fashion, by X:

P_{X}(B) = P(X^{−1}(B)), B ∈ 𝒮 .

Note that P_{X} depends on two functions: the function P and the function X.

EXERCISE 9. Show that if (Ω, ℱ , P) is a probability space and X : (Ω, ℱ) → (S, 𝒮) a random variable then (S, 𝒮 , P_{X}) is a probability space. Hence if X is a random vector then X_{1} is a random variable.

EXERCISE 10. With the notation of Exercise 1, show that the law of H^{∘}X as a random variable on the probability space (Ω, ℱ , P) is the same as the law of H as a random variable on (S, 𝒮 , P_{X}).

So the role of a random variable is to transform an abstract probability space into a concrete one.

In practise, one is often given^{c} a probability measure Q on some (S, 𝒮) and one may (or may not) want to construct a probability space (Ω, ℱ , P) and a random variable X : (Ω, ℱ) → (S, 𝒮) such that P_{X} = P. In the absence of any further requirement we follow Occam’s razor and consider the so-called canonical construction: take Ω = S, ℱ = 𝒮, P = Q and let X(ω) ≡ ω. Then, obviously, P_{X} = Q. If this appears to be silly then this is because it is. But silly things are often quite useful. On the other hand, if other requirements are needed to be satisfied, choosing the “right” probability space and the “right” random variable is an art. The freedom in the choice of probability space can be roughly compared to the freedom of choice of an appropriate coördinate system in ℝ^{3} when dealing, e.g. with the solution of a certain physical problem expressed in terms of a partial differential equation. For example, when we study the motion of fluid in a cylinder we may want to choose cylindrical coördinates.

3.6 Law of a discrete random variable

A discrete random variable X : (Ω, ℱ) → (S, 𝒮) is, by definition, one that takes countably many values. In other words, if P is a probability on (Ω, ℱ) then X is discrete if and only if there is a countable set D ∈ 𝒮 such that P(X ∈ D) = 1. We also assume that D and all its subsets are members of 𝒮. Hence the law P_{X} of X is a probability on D. We know that a probability on a countable set D can be defined by defining its values on singletons. These values form the so-called (in baby probability talk) probability mass function (see 1.1.5 ) Thus, the probability mass function is

p(x) = P_{X}{x} = P(X = x), x ∈ D.

Clearly, if B ⊂ D then

P_{X}(B) = P_{X}( ⋃_{x∈B}{x}) = ∑_{x∈B} p(x).

So p is sufficient for computing P_{X}.

Example 3.1. A box contains n socks labelled 1 through n. Pick two socks at random and let X be the pair of their labels. Then X is a discrete random variable taking values (i, j) where 1 ≤ i, j ≤ n, i ≠ j. The probability mass function is

p(i, j) = 1/n(n − 1).

But what is the probability space? First, we should realise that we do not necessarily need to consider it. Second, we should realise that the article “the” is wrong, for there are many choices for a probability space. Having said that, a reasonable choice for Ω is the set of permutations ω of {1, …, n}, i.e. ω = (ω_{1}, …, ω_{n}) where all the ω_{i} take values in {1, …, n} and are distinct. The σ-field ℱ should be rich enough to contain all singletons, i.e. ℱ = 2^{Ω}. The probability P should be defined as P({ω}) = 1/n! for all ω. To be honest we should check that p(i, j) = P{ω ∈ Ω : ω_{1} = i, ω_{2} = j}. We have

cActually, one is seldom given anything. Either one derives something from some basic principles/requirements or one performs an experiment whereby measurements are collected and a probability measure is stipulated. The latter is the subject of Statistics

P\{ω ∈ Ω : ω_{1} = i, ω_{2} = j\} = \frac{1}{n!} ♯\{ω ∈ Ω : ω_{1} = i, ω_{2} = j\} = \frac{(n − 1)!}{n!} = \frac{1}{n(n − 1)},

as required. So (Ω, ℱ , P) is a probability space, X is a random variable (check this!), P_{X} is its law, and p is its probability mass function.

Example 3.2. Consider the experiment of picking k integers at random (with replacement) from the set {1, …, n}. Our (Ω, ℱ) here can be taken to be

Ω = {1, …, n}^{k}, ℱ = 2^{Ω},

and the probability corresponding to the experiment should be defined as

P\{(ω_{1}, …, ω_{k})\} = \frac{1}{n^{k}} .

Note here that {(ω_{1}, …, ω_{k})} is a set containing a single point, namely the ordered k-tuple ω = (ω_{1}, …, ω_{k}). For more general sets B we take, as required, P(B) = ∑_{ω∈B} P\{ω\} = \frac{♯B}{n^{k}}. Define a random variable representing the maximum number picked:

X(ω) = max(ω_{1}, …, ω_{k}).

Clearly, X takes values in {1, …, n} and trivially, X is a measurable function from (Ω, ℱ) to ({1, …, n}, 2^{{1,...,n}}). (The inverse image of a subset of {1, …, n} under X is, of course, a subset of Ω, i.e. an element of ℱ.) Let us compute the probability mass function for X. For x ∈ {1, …, n} we have

p(x) = P(X^{−1}(x)) = P(X = x) = P{ω ∈ Ω : max(ω_{1}, …, ω_{k}) = x}

= P{ω ∈ Ω : max(ω_{1}, …, ω_{k}) ≤ x but max(ω_{1}, …, ω_{k}) ≰ x − 1}

= P{ω ∈ Ω : max(ω_{1}, …, ω_{k}) ≤ x} − P{ω ∈ Ω : max(ω_{1}, …, ω_{k}) ≤ x − 1}

= \frac{1}{n^{k}} ♯\{ω ∈ Ω : ω_{1} ≤ x, …, ω_{k} ≤ x\} − \frac{1}{n^{k}} ♯\{ω ∈ Ω : ω_{1} ≤ x − 1, …, ω_{k} ≤ x − 1\}

= \frac{x^{k} − (x − 1)^{k}}{n^{k}} .

EXERCISE 11. Suppose that the random variable X takes n distinct values (i.e. X(Ω) is a set with n elements). Show that σ(X) has 2^{2^{n}} elements and describe (give a procedure for describing) them.

3.7 Tossing coins

Consider the experiment of flipping fair coins independently (see 2.2 ). A coin takes 2 values, heads or tails with probability 1/2 each. Label heads by 1 and tails by 0. Let ω_{1} be outcome of the first toss, ω_{2} that of the second, etc. We stipulate that:

A coin flipped n times yields a string (ω_{1}, …, ω_{n}) of n 0s or 1s with probability 1/2^{n} each.

For example (0, 1, 0) should have probability 1/8, while (1, 1, 0, 0, 1) should have probability 1/32. In other words, we are trying to define a probability P by saying that the P of an event that specifies the first n outcomes completely should be 1/2^{n}, for each n ∈ ℕ. Consider our desire to find out when we will get our first 1 (head), i.e. let

T = inf{n ∈ ℕ : ω_{n} = 1}.

This appears to be a random variable. But on which measurable space? We cannot pick Ω to be {0, 1}^{n} for some finite n, because we have no bound on T. We can, however, pick

Ω = {0, 1}^{ℕ},

i.e. the set of all infinite-length sequences ω = (ω_{1}, ω_{2}, …), where ω_{n} = 0 or 1 for all n. This is our coin-flip space.

EXERCISE 12. Show that the coin-flip space {0, 1}^{ℕ} is uncountable.

To put what we said in some notation, let

R(i_{1}, …, i_{n}) := {ω ∈ Ω : ω_{1} = i_{1}, …, ω_{n} = i_{n}}.

To this set we should assign probability 2^{−n}:

prob(R(i_{1}, …, i_{n})) = 2^{−n}.  (3.3)

We use the symbol prob rather than P because we have not yet verified that this is a probability. Let ℛ be the class of such sets (together with the empty set):

ℛ := {∅} ∪ {R(i_{1}, …, i_{n}) : i_{1}, …, i_{n} ∈ {0, 1}, n ∈ ℕ}.

Formula (3.3) defines the function

prob : ℛ → ℝ.

It is by no means true that ℛ contains all subsets of Ω (far from that):

EXERCISE 13. Show that

R(1) ∪ R(0, 1) ∪ R(0, 0, 1) ∪ ⋯ ∉ ℛ.

Show that this set is the set {ω ∈ Ω : T (ω) < ∞}. Show that the intersection of two elements of ℛ is in ℛ, but not the union.

EXERCISE 14. Second, the complement of any set in ℛ is written as a finite disjoint union of sets in ℛ. (For example R(0, 1)^{c} = {ω : ω_{1} = 1} ∪ {ω : ω_{2} = 0} = R(1) ∪ {ω : ω_{2} = 0} = R(1) ∪ R(1, 0) ∪ R(0, 0).)

What kind of properties does prob have? We would like prob to be, at least, additive, i.e. to satisfy prob(G_{1} ∪ ⋯ ∪ G_{k}) = prob(G_{1}) + ⋯ + prob(G_{k}) if G_{1}, … G_{k} are mutually disjoint elements of ℛ. There is an issue here: as we saw, the union of elements of ℛ may not be in ℛ. So this last additivity property is completely meaningless for prob. We remedy this by enlarging ℛ:

𝒞 := {G_{1} ∪ ⋯ ∪ G_{k} : G_{1}, …, G_{k} are mutually disjoint elements of ℛ, k ∈ ℕ},

and by extending prob to 𝒞 in the most natural way:

P_{0}(G_{1} ∪ ⋯ ∪ G_{k}) := ∑_{j=1}^{k} prob(G_{j}).

Lemma 3.9. (i) 𝒞 is a field. (ii) P_{0} is well-defined and additive over 𝒞.

Proof (i) Let A, B ∈ 𝒞. Then A = ∪_{i=1}^{n}G_{i}, B = ∪_{j=1}^{m}H_{j}, where the G_{i} are mutually disjoint (m.d.) elements of 𝒞, and so are the H_{j}. We have

AB = ∪_{i,j}G_{i}H_{j}

and, clearly, the G_{i}H_{j} are m.d. elements of ℛ; so AB ∈ 𝒞. Also,

A^{c} = ∩_{j}G_{i}^{c},

each G_{i}^{c} is an element of 𝒞 (Exercise 14), and because we just proved that 𝒞 is closed under intersections, we have A^{c} ∈ 𝒞. So 𝒞 is a field. (ii) To show that P_{0} is well-defined means to show that if we can write the same A ∈ 𝒞 in two different ways, say A = ∪_{i=1}^{n}G_{i} = ∪_{j=1}^{m}H_{j} where both the G_{i} and the H_{j} are m.d. elements of ℛ, we have ∑_{i} prob(G_{i}) = ∑_{j} prob(H_{j}). But notice that G_{i} = ∪_{j}G_{i}H_{j} and so prob(G_{i}) = ∑_{j} prob(G_{i}H_{j}), and, similarly, prob(H_{j}) = ∑_{i} prob(G_{i}H_{j}). This proves the claim that P_{0} is well-defined on 𝒞. Now, to show additivity, suppose that A_{1}, …, A_{n} are m.d. elements of 𝒞. We must show that P_{0}(∪_{i}) = ∑_{i} P_{0}(A_{i}). Now, to show additivity, suppose that A_{1}, …, A_{n} are m.d. elements of 𝒞. We must show that P_{0}(∪_{i}A_{i}) = ∑_{i} P_{0}(A_{i}). But each A_{i} is itself a finite union of m.d. elements of ℛ, say A_{i} = ∪_{j}H_{ij}, and, by the definition of P_{0}, we have P_{0}(A_{i}) = ∑_{j} prob(H_{ij}). So ∑_{i} P_{0}(A_{i}) = ∑_{i} ∑_{j} prob(H_{ij}). On the other hand, ∪_{i}A_{i} = ∪_{i,j}H_{ij} and the H_{ij} are m.d. elements of ℛ. Hence, by the definition of P_{0} again, we have P_{0}(∪_{i}A_{i}) = ∑_{i} ∑_{j} prob(H_{ij}). □

To put in words, what we have shown so far is that, starting merely from the intuitive definition (3.3), we can compute probabilities of sets that can be expressed using finitely many outcomes, i.e. that involve ω_{i} up to some finite index. Mathematically, Lemma 3.9 has enabled us to define a function

P_{0} : 𝒞 → ℝ

in such a way that

P_{0}(G) = prob(G), if G ∈ ℛ

and

P_{0}(G_{1} ∪ ⋯ ∪ G_{k}) = P_{0}(G_{1}) + ⋯ + P_{0}(G_{k}), if G_{1}, … G_{k} are mutually disjoint elements of ℛ

(a property which, as remarked, is meaningless for prob).

But there are many sets, whose probability we would like to know, but which do not belong to 𝒞.

Example 3.3. The following (useful) sets do not belong to 𝒞:

(i) A_{1} := {ω : T (ω) < ∞} (i.e. the event that a head will occur at some point). Indeed, this equals ∪_{n=1}^{∞}{ω ∈ Ω : ω_{n} = 1} and does not belong to 𝒞 because it involves infinitely many coordinates. (ii) A_{2} := {ω : ω_{n} = 1 for infinitely many n}. This, obviously, does not belong to 𝒞. Formally the set equals

{ω : ∀n ∃m ≥ n ω_{m} = 1} = ⋂_{n} ⋃_{m≥n}{ω : ω_{m} = 1} = ⋂_{n} ⋃_{m≥n} ⋃_{i_{1},...,i_{m}∈{0,1}} R(i_{1}, …, i_{m−1}, 1).

And so it is explicitly seen that we need infinitely many sets from ℛ to construct A_{2}. Although we feel that both A_{1} and A_{2} should be assigned probability 1, we cannot write P_{0}(A_{1}) = 1, neither P_{0}(A_{2}) = 1, simply because neither of the two sets belongs to the domain of the function P_{0}.

So our desire to deal with these sets (and not only!) forces us to carry the story further. We saw (Lemma 3.9 that P_{0} is additive on 𝒞. But more is true:

Lemma 3.10. P_{0} is countably additive over 𝒞, that is, if G_{n} are mutually disjoint elements of 𝒞 and if ∪_{n}G_{n} ∈ 𝒞 then P_{0}(∪_{n}G_{n}) = ∑_{n} P_{0}(G_{n}).

Proof By Lemma [CA], Appendix B, it suffices to show that if G_{n} is a decreasing sequence of elements of 𝒞 such that ∩_{n}G_{n} = ∅ then P_{0}(G_{n}) → 0. – To be continued – □

We next consider the class D(𝒞) of sets that are limits of sequences of sets in 𝒞. Lemma [FIRST EXTENSION], Appendix B, shows that we can define P_{1} on D(𝒞) such that P_{1} is countably additive and agrees with P_{0} on 𝒞.

We then consider the class D^{2}(𝒞) = D(D(𝒞)) of sets that are limits of sequences of sets in D(𝒞). Just as above, we can define P_{2} on D^{2}(𝒞) such that P_{2} is countably additive and agrees with P_{1} on D(𝒞).

One would suspect that by continuing, in this manner, by induction, i.e. by taking sets which are limits of limits of limits, ad infinitum, we would exhaust the class

ℱ = σ-field generated by 𝒞 .  (3.4)

(It is not difficult to see that, also,

ℱ = σ-field generated by ℛ.)  (3.5)

Unfortunately, this does not work. The way out is to do as in Proposition [SANDWICH], Appendix B. This proposition actually proves the following:

Theorem 3.1. Let Ω = {0, 1}^{ℕ}. Let ℛ be the class of sets of the form R(i_{1}, …, i_{n}) := {ω ∈ Ω : ω_{1} = i_{1}, …, ω_{n} = i_{n}}. Define prob(R(i_{1}, …, i_{n})) = 2^{−n}. Let ℱ be the σ-field generated by ℛ. Then there exists a UNIQUE probability P on (Ω, ℱ) such that P(G) = prob(G) for every G ∈ ℛ.

Proof See proof of Proposition [SANDWICH], Appendix B. To prove uniqueness, assume that there is another probability P̃ on (Ω, ℱ) that agrees with prob on ℛ. Then P̃ agrees with prob (and hence with P on 𝒞. Let us define the class common domain of P, P̃:

𝒟 := {A ∈ 2^{Ω} : P(A) = P̃(A)}.

We have

𝒞 ⊂ 𝒟.

Therefore, of λ(𝒞) (resp. λ(𝒟)) is the smallest λ-system containing 𝒞 (resp. 𝒟) we have

λ(𝒞) ⊂ λ(𝒟).

But it is easy to see that 𝒟 is itself closed under proper differences and increasing limits. Therefore λ(𝒟) = 𝒟. On the other hand, by Lemma [Sierpiński-Dynkin], Appendix B, we have λ(𝒞) = σ(𝒞). So ℱ := σ(𝒞) is contained in 𝒟. In other words, the two probabilities agree on ℱ. □

This theorem tells us that

({0, 1}^{ℕ}, ℱ , P) is a probability space.

So we can use all the good things we know about a probability space. For example, T (ω) = inf{n ∈ ℕ : ω_{n} = 1} defines a measurable function T : (Ω, ℱ) → (ℝ, ℬ) and, since {T ≤ n} are increasing sets with union equal to the set {T < ∞}, we have P(T < ∞) = lim_{n→∞} P(T ≤ n). But {T > n} = R(0, …, 0) (n 0’s inside the parenthesis) and so P(T > n) = prob(R(0, …, 0)) = 2^{−n}. Thus P(T < ∞) = lim_{n}(1 − 2^{−n}) = 1.

EXERCISE 15. Consider the space Ω = {0, 1}^{ℕ} and the class of sets ℛ defined in the tossing coins section. (i) Show that ℛ is a countable set, i.e. there is a one-to-one correspondence between ℛ and ℕ. (ii) Show that 𝒞 is the field generated by ℛ and is also countable. (iii) Consider ℱ, the σ-field generated by ℛ (and also by 𝒞) and show that it is uncountable; more specifically, show that there is a one-to-one correspondence between ℱ and 2^{ℕ}. (iv) Consider the class 2^{Ω} of all subsets of Ω and show that there is a one-to-one correspondence between 2^{Ω} and 2^{2^{ℕ}} (the class of subsets of the set of subsets of ℕ.) It is known [3] that the cardinality of the set of subsets of a set is strictly larger than the cardinality of the set itself (Schroeder-Bernstein theorem). Using this, conclude that there 2^{Ω} is strictly larger than ℱ. In fact MUCH larger: In some sense, most subsets of Ω are outside ℱ! This means that most sets of sequences of coin tosses cannot be expressed using countably many operations on elementary sets (i.e. sets from ℛ). The CURIOUS thing though is that we know very few of these sets: although the majority of them are outside ℱ, somehow, it is very heard to come up with an example.^{d}

3.8 Uniform random variable

We will now construct our first non-discrete random variable. Consider the probability space (Ω = {0, 1}^{ℕ}, ℱ , P) of the coin tossing experiment. Define

U(ω) = \frac{ω_{1}}{2} + \frac{ω_{2}}{2^{2}} + \frac{ω_{3}}{2^{3}} + ⋯ = ∑_{k=1}^{∞} \frac{ω_{k}}{2^{k}} .  (3.6)

Let U_{n}(ω) := ∑_{k=1}^{n} \frac{ω_{k}}{2^{k}}: these are simple random variables. Since U(ω) = lim_{n→∞} U_{n}(ω), we have, by Lemma 3.3, that U : (Ω, ℱ) → (ℝ, ℬ) is measurable (i.e. a random variable). It is ONE random variable because it takes values in ℝ. More specifically, observe that

0 ≤ U(ω) ≤ ∑_{k=1}^{∞} \frac{1}{2^{k}} = 1.

So U(Ω) ⊂ [0, 1]. Our first observation is

Lemma 3.11. For any integer 0 ≤ a < 2^{n} and any n ∈ ℕ, unless ω_{i} = 1 for all i > n,

U ≥ \frac{a}{2^{n}} ⟺ U_{n} ≥ \frac{a}{2^{n}} .

Proof Note that 2^{n}U_{n} is an integer. If it is not the case that ω_{i} = 1 for all i > n we have 2^{n}U − 2^{n}U_{n} < 1, i.e. the integer part of 2^{n}U equals 2^{n}U_{n} and hence 2^{n}U ≥ a iff 2^{n}U_{n} ≥ a. □

Next, we compute the following probabilities:

Lemma 3.12. For any integer 0 ≤ a < 2^{n} and any n ∈ ℕ, we have

P(U ≤ a/2^{n}) = a/2^{n}.

dIn some sense, this exercise is a baby version of Gödel’s theorem which says, naively speaking, that most theorems cannot be proved. Except that we hardly ever encounter them.

Proof Using Lemma 3.11 we find that

P(U < a/2^{n}) = P(U_{n} < a/2^{n}).

We use induction on n. To start, let n = 1. Observe that P(U_{1} < 1/2) = P(ω_{1} = 0) = 1/2, as required. Now assume the statement to be true up to index n − 1. Write

U_{n}(ω) = \frac{ω_{1}}{2} + \frac{1}{2}Y_{n−1}(ω),
Y_{n−1}(ω) = \frac{ω_{2}}{2} + \frac{ω_{3}}{2^{2}} + ⋯ + \frac{ω_{n−1}}{2^{n−1}} .

Therefore

P(U_{n} < a/2^{n}) = P(ω_{1} = 0, Y_{n−1} < a/2^{n−1}) + P(ω_{1} = 1, Y_{n−1} < a/2^{n−1} − 1)
= \frac{1}{2}P(Y_{n−1} < a/2^{n−1}) + \frac{1}{2}P(Y_{n−1} < a/2^{n−1} − 1).

By the induction hypothesis, P(Y_{n−1} < c/2^{n−1}) = c/2^{n−1} for all c < 2^{n−1} (why?). If a ≤ 2^{n−1}, the second term in the display is 0 and the first equals (1/2) × (a/2^{n−1}) = a/2^{n}, as needed. If 2^{n−1} < a < 2^{n}, the first term equals 1/2 and the second equals (1/2) × (a/2^{n−1} − 1); the sum gives again a/2^{n}. □

Finally, we have

Lemma 3.13. For any real number x ∈ [0, 1],

P(U < x) = P(U ≤ x) = x.

Proof Let [y] denote the largest integer not exceeding the real number y. Define, recursively,

z_{1} = x, ξ_{1} = [2z],

z_{n+1} = z_{n} − \frac{ξ_{n}}{2^{n}} , ξ_{n+1} = [2^{n+1}z_{n+1}].

We have 2^{n}z_{n+1} = 2^{n}z_{n} − ξ_{n} = 2^{n}z_{n} − [2^{n}z_{n}] ≤ 1, so z_{n+1} ≤ 1/2^{n} for all n. Therefore the sequence z_{n} converges to 0 and the numbers ξ_{n} are either 0 or 1. We also have

x_{n} := \frac{ξ_{1}}{2} + \frac{ξ_{2}}{2^{2}} + ⋯ + \frac{ξ_{n}}{2^{n}} = (z_{1} − z_{2}) + (z_{2} − z_{3}) + ⋯ + (z_{n} − z_{n_{+}1}) = x − z_{n+1},

and so x_{n} → x, as n → ∞. Also, x_{n} is of the form a/2^{n}, where a is an integer. So P(U ≤ x_{n}) = x_{n}. Since x_{n} actually increases to x, we have ∪_{n=1}^{∞}{U ≤ x_{n}} = {U < x}, and so taking limits we get P(U < x) = x, for all x ≤ 1. This also implies P(U ≤ x + 1/n) = x + 1/n for all n sufficiently large. But ∩_{n}{U ≤ x + 1/n} = {U ≤ x} and so, taking limits, P(U ≤ x) = x. □

Corollary 3.2.

P(U = x) = 0, for all x ∈ [0, 1].

So the random variable U takes no specific value with positive probability. Yet, U exists (we constructed it). This is curious, but we can (must) get used to it. Any real random variable with this property is called continuous random variable. In books, U has a name: it is called uniform random variable on the interval [0, 1].

Corollary 3.3. Let ℚ be the set of all rational numbers in [0, 1]. Then

P(U ∈ ℚ) = 0.

Proof The set ℚ is enumerable (has countably many points). Hence

P(U ∈ ℚ) = P(∪_{q∈ℚ}{U = q}) = ∑_{q∈ℚ} P(U = q) = 0.

□

3.9 Probability, revisited

Recall, once more, that a probability P defined on a σ-field ℱ of subsets of a set Ω is a function P : ℱ → ℝ such that (i) P(Ω) = 1, and (ii) P(∪_{n}A_{n}) = ∑_{n} P(A_{n}), if the A_{n} are mutually disjoint elements of ℱ.

If Ω is a countable set then we know that it suffices to define P on singletons, i.e. knowledge of P{ω} for all ω ∈ Ω implies knowledge of P(A) for all A ∈ ℱ. Indeed,

P(A) = ∑_{ω∈A} P{ω}.

We saw, in the previous section, the necessity to define P on sets which are not countable. Can’t we define P{ω} for all ω in this case? The argument is simple. Suppose we did do so. Let Ω_{+} := {ω ∈ Ω : P{ω} > 0}, and, for each n ∈ ℕ, Ω_{n} = {ω ∈ Ω : P{ω} ≥ 1/n}. This is a finite set with at most n points. Observe that ∪_{n}Ω_{n} = Ω_{+}. But this tells us that Ω_{+} is a countable set. So this poses a “philosophical” problem: If we want to define a random variable that is uniformly distributed on the uncountable set [0, 1] by defining probabilities of singletons, then we’d have to pick a countable subset of [0, 1] and define probabilities of points there. But which one to pick? We cannot give preference to any particular countable subset.

The way out is not to define probabilities on single points, but, rather, on sets, as we did in the previous section. We then do not give preference to any particular countable subset, but we lose something “intuitive”: we assign zero probability to every individual point. This is what needs to be done, both for physical and mathematical reasons, and this is what shall be done hereafter: We will be defining probabilities on sets.

But another burden appears now: whereas, in the countable case, we were free to define probabilities of points at will (as long as they summed up to something finite), in the general case we cannot define probabilities of sets at will: we must respect the countable additivity property (ii) above. Since there are far too many sets, we will try to see how to define P on a class of selected few of them. So read the next section.

3.10 Distribution functions

Consider a real random variable X : (Ω, ℱ) → (ℝ, ℬ) and put a probability P on Ω. Recall the definition of the law P_{X} of X:

P_{X}(B) = P(X ∈ B), B ∈ ℬ.

Up until the middle of the 19th century, a function was an object that was given by a formula. Nowadays, of course, a function is more abstract: we certainly need no formulae to comprehend what a function is. However, let us take the old-fashioned point of view. Clearly, P_{X} is a function whose arguments are sets (elements of ℬ). These are a lot of arguments! And, as we know, we don’t need all of them because, for example, if B = B_{1} ∪ B_{2} and B_{1} ∩ B_{2} = ∅, then P_{X}(B) = P_{X}(B_{1}) + P_{X}(B_{2}). To put it pedantically, we can’t define P_{X}(B) anyway we like. We must make sure that it is consistent (countable additivity must hold).

So the question is: can we throw away some (many) of the values P(B) without losing information?

The answer is yes. It suffices to know P_{X}(B) only for B of specific form. There are many choices. One choice is to take

B = (−∞, x],

where x ranges from −∞ to ∞ and consider only (see Example 1.5 )

F (x) := P_{X}(−∞, x], x ∈ ℝ.  (3.7)

So I claim that:

Lemma 3.14. Knowledge of P_{X} on this class of sets only (semi-infinite intervals) implies knowledge of P_{X} on the whole of ℬ.

This is a big claim. But we shall prove it by construction. First, let us see the properties of F.

Lemma 3.15. (i) x_{1} < x_{2} ⇒ F (x_{1}) ≤ F (x_{2}), (ii) lim_{x→−∞} F (x) = 0, (iii) lim_{x→+∞} F (x) = 1, (iv) lim_{n→∞} F (x + 1/n) = F (x).

Proof (i) If x_{1} < x_{2} we have (−∞, x_{1}] ⊂ (−∞, x_{2}] and so P_{X}(−∞, x_{1}] ≤ P_{X}(−∞, x_{2}]. (ii)&(iii) Since F is monotonic with values between 0 and 1 it has a limit both as x tends to −∞ and +∞. We have ∩_{n∈ℤ}(−∞, n] = ∅, so P_{X}(−∞, n] → 0 as n → −∞. Similarly, ∪_{n∈ℤ}(−∞, n] = ℝ, so P_{X}(−∞, n] → P_{X}(ℝ) = 1 as n → +∞. (iv) ∩_{n}(−∞, x + 1/n] = (−∞, x]. □

Remarks:

1. Notice that F (x − 1/n) does NOT necessarily converge to F (x) because ∪_{n}(−∞, x − 1/n] = (−∞, x), and so lim_{n→∞} F (x − 1/n) = P_{X}(−∞, x) which will be different from P_{X}(−∞, x] if P_{X}{x} > 0.

2. There is no good reason that we chose this class of sets, other than people have been using it by convention. For instance, we could have chosen open semi-infinite interval (−∞, x), in which case (iv) of Lemma 3.15 would be replaced by lim_{n→∞} F (x − 1/n) = F (x). Proof of Lemma 3.14 Consider the coin-flip Ω = \{0, 1\}^{ℕ}. U(ω) = ∑_{n=1}^{∞} \frac{ω_{n}}{2^{n}}. Then U is a random variable and P is, indeed, a probability on (Ω, ℱ), where ℱ is defined through (3.4) or (3.5). We also showed that P(U ≤ u) = u for all u ∈ [0, 1]. Suppose now that F, as in (3.7), is given. Define

F^{−1}(u) := sup{x ∈ ℝ : F (x) ≤ u}.  (3.8)

By Lemma 3.3, F^{−1}(U) is a random variable. In other words, F^{−1}^{∘}U : (Ω, ℱ) → (ℛ, ℬ) is measurable. Suppose that, for some u, t, we have F^{−1}(u) ≤ t. This means that t is an upper bound of the set {x ∈ ℝ : F (x) ≤ u}. Hence for all x ∈ ℝ, if F (x) ≤ u then x ≤ t. Equivalently, for all x ∈ ℝ, if x > t then F (x) > u. Equivalently, for all ε > 0, F (t + ε) > u. Equivalently, for all n ∈ ℕ, F (t + 1/n) > u. Thus,

{ω ∈ Ω : F^{−1}(U(ω)) ≤ t} = {ω ∈ Ω : ∀n ∈ ℕ F (t + 1/n) > U(ω)}.

Hence the probabilities of the two sets (notice they both belong to ℱ) are the same:

P(F^{−1}(U) ≤ t) = P(∀n ∈ ℕ F (t + 1/n) > U).

By the fact that P(A_{n}) → P(A) if A_{n} is a decreasing sequence of sets with intersection A, we have

P(∀n ∈ ℕ F (t + 1/n) > U) = lim_{n→∞} P(U < F (t + 1/n)).

By Lemma 3.13, and the right-continuity of F (Lemma 3.15),

lim_{n→∞} P(F (t + 1/n) > U) = lim_{n→∞} F (t + 1/n) = F (t).

Thus the random variable F^{−1}^{∘}U has the same law as X. Hence, for all B ∈ ℬ,

P_{X}(B) = P(F^{−1}(U) ∈ B) = P{ω ∈ {0, 1}^{ℕ} : F^{−1}(U(ω)) ∈ B}

So, knowledge of P_{X} on sets of the form (−∞, x] implies knowledge of P_{X}(B) for all B ∈ ℬ by the explicit formula of the last display. □

Definition 3.1. A function F : ℝ → ℝ is called distribution function iff (i) x_{1} < x_{2} ⇒ F (x_{1}) ≤ F (x_{2}), (ii) lim_{x→−∞} F (x) = 0, (iii) lim_{x→+∞} F (x) = 1, (iv) lim_{n→∞} F (x + 1/n) = F (x).

Corollary 3.4. If F is a distribution function then there exists a probability Q on (ℝ, ℬ) such that Q(−∞, x] = F (x) for all x ∈ ℝ.

Proof Let Ω = {0, 1}^{ℕ}. Let ℱ be as in (3.4) or (3.5). Let P be the probability on (Ω, ℱ) as in Theorem 3.1. Let U : (Ω, ℱ) → (ℛ, ℬ) be the random variable defined by U(ω) = ∑_{n=1}^{∞} 2^{−n}ω_{n}. Let F^{−1} be defined by (3.8). Let X = F^{−1}^{∘}U. Then Q = P_{X}. □

Lemma 3.16. Let X be one real random variable with law P_{X} and distribution function F (x) = P_{X}(−∞, x], x ∈ ℝ. Then (i) P(X ∈ (a, b]) = F (b) − F (a), (ii) P(X ∈ (a, b)) = F (b−) − F (a), (iii) P(X ∈ [a, b]) = F (b) − F (a−), (iv) P(X = a) = F (a) − F (a−).

Proof (i)

(a, b] = (−∞, b] − (−∞, a].

(ii)

(a, b) = ∪_{n∈ℕ}(a, b − 1/n]

(iii)

[a, b] = ∩_{n∈ℕ}(a + 1/n, b].

(iv)

{a} = [a, a].
□

EXERCISE 16. Carefully justify the formulae in the proof of Lemma 3.16.

EXERCISE 17. There is nothing kosher about choosing F so that it is right continuous. It is, merely, a convention. Another choice could be F (x) = P(−∞, x), which (show this) results into a left continuous function. Yet another choice is to take this last function and modify it at each of points of discontinuity x and giving it the value \frac{1}{2}(F (x−) + F (x+)).

3.11 Types of distribution functions

In this section we discuss the various kinds of distribution functions on ℝ.

3.11.1 Discrete distribution functions

A discrete distribution function is the distribution function of a discrete random variable X with values in some countable subset S of ℝ. Assume that p(s) = P(X = s) > 0 for all s ∈ S. Such a distribution function satisfies F (s)−F (s−) > 0 for all s ∈ S. Indeed, F (s)−F (s−) = P(X = s). Also, if (a, b) is an open interval containing no points of S, then F is constant on (a, b). Indeed, if a < x < b then F (x) − F (a) = P(a < X ≤ x) = ∑_{s∈S,a<s≤x} P(X = s) = 0.

Example 3.4. Let X be a random variable such that P(X = n) = 2^{−n}, n ∈ ℕ. Then its distribution function looks like Example 3.5. Let X be a random variable such that for every rational number of the form m/n where m, n are integers with no common factors, we have P(X = m/n) = c2^{−(m+n)} where c is chosen so that P(X ∈ ℚ) = 1. Its distribution function is discrete because ℚ is countable. Unfortunately, I can’t draw it. (There are no intervals (a, b) containing no rational points.)

3.11.2 Continuous distribution functions

A distribution function F is continuous if it is a continuous function, i.e. if F (x) − F (x−) = 0 for all x ∈ ℝ.

Example 3.6. Consider the random variable U with P(U ≤ u) = u for all u ∈ [0, 1]. Such a random variable exists (we constructed it). Its distribution function looks like

Excepting the points 0, 1 we have that it is also differentiable with derivative f(u) = 1 if 0 < u < 1 and 0 otherwise.. If we arbitrarily define f(0) = f(1) = 0, we also have ∫_{−∞}^{u} f(t)dt = F (u) for all u ∈ ℝ. We like such distribution functions:

Absolutely continuous distribution functions

A distribution function F is called absolutely continuous if there exists a function f (called density of F) such that

F (x) = ∫_{−∞}^{x} f(t)dt, x ∈ ℝ.

e The density is not uniquely defined. For instance, it can be changed on a finite set and such a change will not affect the integral above. Usually, one^{f} imposes additional regularity conditions, such as continuity, resulting in uniqueness.

But not all continuous distribution functions are absolutely continuous:

Singularly continuous distribution functions

A distribution function F is called singularly continuous if it is continuous but not absolutely continuous. We need to show that there are such functions.

Example 3.7. Consider the coin-flip space (Ω = {0, 1}^{ℕ}, ℱ , P) and let

V (ω) := ∑_{n=1}^{∞} \frac{2ω_{n}}{3^{n}} .

eThe integral in the display is a Lebesgue integral. For a definition, skip to §3.13.3. For the time being, you may think of it as the standard Riemann integral of Integral Calculus.

f unconsciously

EXERCISE 18. Show that the random variable V defined in Example 3.7 has a continuous but not absolutely continuous distribution function.

3.11.3 General distribution functions

Suppose that F, G are distribution functions. Then, for any λ ∈ (0, 1), the function λF +(1−λG) is a distribution function. (Probabilistically, if X, Y are random variables with distribution functions F, G, respectively, then we can define a new random variable Z which equals X with probability λ or Y with probability 1−λ.) So, if F is discrete and G continuous then λF +(1−λG) is neither discrete nor continuous: it is mixed. The question is: Can we exhaust all distribution functions by taking mixtures of the three types mentioned above? The answer is yes:

Theorem 3.2. Let F be a distribution function on ℝ. Then F can be uniquely written as

F = λ_{d}F_{d} + λ_{ac}F_{ac} + λ_{sc}F_{sc}

where F_{d}, F_{ac}, F_{sc} are discrete, absolutely continuous, singularly continuous distribution functions, respectively, and where the coefficients are nonnegative such that λ_{d} + λ_{ac} + λ_{sc} = 1.

The last two terms of this decomposition are known as the continuous part of F. The first two terms are known as the singular part of F. We will not prove this theorem, but refer, e.g. to [2].

3.11.4 Differentiation: a word of caution

The subject of densities involves the concept of a derivative of functions that are not necessarily everywhere differentiable. Mimicking the definition of a density, we will say that a function G has density g if G′(x) = g(x) for almost all x. The latter statement means that it holds true that G′(x) = g(x) for all x in some set A whose complement is small in the sense that for all ε > 0 there exist intervals I_{n}, n ∈ ℕ with lengths λ_{n}, n ∈ ℕ, such that ∑_{n} λ_{n} < ε and A^{c} ⊂ ∪_{n}I_{n}. We take this latter statement as a definition:

Definition 3.2. (i) We say that a set B ⊂ ℝ has measure zero if all ε > 0 there exist intervals I_{n}, n ∈ ℕ with lengths λ_{n}, n ∈ ℕ, such that ∑_{n} λ_{n} < ε and B ⊂ ∪_{n}I_{n}. (ii) A function G : ℝ → ℝ is said to be almost everywhere differentiable if G′(x) exists for all x ∈ A where A^{c} has measure zero.

We can verify the following:

• Any countable set has measure zero.

• A set B ⊂ [0, 1] has measure zero if and only if a uniform random variable U in [0, 1] satisfies P(U ∈ B) = 0.

• An absolutely continuous distribution function is almost everywhere differentiable.

What kind of functions G are almost everywhere differentiable with a derivative g that can be used to recover the function? That is, for what kind of functions G can we apply the Fundamental Theorem of Calculus (FTC), i.e. the statement that G(b) − G(a) = ∫_{a}^{b} g(x)dx? Calculus tells us that if G is piecewise continuously differentiable then this is true. However, we saw, by means of distribution functions, not only that the FTC holds more generally but it is desirable to understand it more generally because random variables with not everywhere differentiable distribution functions abound in theory and in practise. Here is a definition.

Definition 3.3. A function G : ℝ → ℝ is called absolutely continuous if for all ε > 0 there exists a δ > 0 such that for any finite collection of disjoint intervals [a_{k}, b_{k}] we have ∑_{k} |G(b_{k}) − G(a_{k})| < ε provided that the sum of the lengths of the intervals is less than δ.

EXERCISE 19. A differentiable function is absolutely continuous. An absolutely continuous function is continuous.

Theorem 3.3. Any absolutely continuous function G is differentiable almost everywhere and if g is its derivative then the FTC holds.

For a proof see [2]. Alternatively, one can use Probability Theory to prove all that, provided that one has understood the theory of Martingales, a subject of a later chapter. Indeed, much of this generalised differentiation theory achieves a beautiful interpretation and becomes comprehensible in probabilistic terms, via Martingale Theory.

EXERCISE 20. Show that a continuous and piecewise differentiable function G is almost everywhere differentiable.

3.12 Transformation rules and densities

Consider a random variable X : (Ω, ℱ) → (ℛ, ℬ). Suppose P is a probability on (Ω, ℱ). We are interested in the distribution P_{X} of X. Suppose, for some reason, we don’t like it and want to change it to something else. There are two ways to do this. First, we can change the probability P and replace it by some other probability Q. Then P_{X} will be replaced by Q_{X}. Second, we can take a function H : (ℛ, ℬ) → (ℛ, ℬ) and replace X by H^{∘}X. Then P_{X} is replaced by P_{H∘X}. The two procedures are not, in general, equivalent.

Suppose, for instance, that X is a discrete random variable. Then any one-to-one function H will not change the probabilities of singletons {s} such that P_{X}{s} > 0, but, merely, will rename them: {s} will be transformed to {H(s)} and its probability will remain the same. Even if H is not one-to-one, there is not much that H can do to change the probabilities. Consider, for instance, a random variable X with values 1, 2, 3 and probabilities p_{1}, p_{2}, p_{3}, respectively. Then the most a function H can do is either be one-to-one, in which case H(1), H(2), H(3) will retain the old probabilities, or map two points, say 1, 2, to a single point, with probability p_{1} + p_{2} and leave the the third intact. Thus, the types of changes in the distribution of a discrete X that can be achieved by taking a function of it are quite restricted. To really change its distribution ad libitum, we need to change the underlying probability P.

For the case of absolutely continuous random variables, the story is different: a merely oneto-one function H can simultaneously change the values and the distribution in a quite general fashion.

Theorem 3.4. Let X be an absolutely continuous random variable in ℝ with density f. Let φ : ℝ → ℝ be strictly increasing differentiable function and let ψ be its inverse function. Then φ(X) a random variable with absolutely continuous distribution function and density

ψ′ · f^{∘}ψ on φ(ℝ),

and 0 elsewhere.

Proof Since φ is strictly increasing, its inverse function exists and has domain φ(ℝ). Then, the distribution function of φ(X) is, for any t ∈ φ(ℝ),

P(φ(X) ≤ t) = P(X ≤ ψ(t)) = ∫_{−∞}^{ψ(t)} f(x)dx.

By changing variable in the integral we have

∫_{−∞}^{ψ(t)} f(x)dx = ∫_{−∞}^{t} f(ψ(s))ψ′(s)ds,

where we set ψ′(s) = 0 for s ∉ φ(ℝ). From the definition of an absolutely continuous distribution function we see that, indeed, φ(X) has absolutely continuous distribution function and its density is the function inside the last integral. □

EXERCISE 21. Let U be a uniform random variable in the interval (0, 1). Find the density function of e^{e^{U}}.

Theorem 3.4 assumes that φ is strictly increasing. It is immediate to find out the formula for strictly decreasing φ. Generalising to more general functions is possible and relatively easy for random variables with values in ℝ (the story in ℝ^{d} is more complicated). For instance, we may assume that φ is piecewise differentiable. The problem becomes a problem in differential calculus and the general theorem is omitted. However, an example is due:

EXERCISE 22. Let X be a random variable with density f(x) = c(1 + x^{2})^{−1}, x ∈ ℝ. Let φ(x) = cosh(x). Find the density (and hence show that it exists) of φ(X).

3.13 Expectation

The expectation of one real random variable is, if it can be defined, an important numerical aspect of the random variable. It is justified, for instance, by the Theorem (Law) of Large Numbers which will be proved at a later chapter.

It is easy to define the expectation of a discrete random variable X with values in ℝ and probability mass function p(x). Let S be the set of x such that p(x) > 0. Then

EX = ∑_{x∈S} xp(x),

provided that this sum can be defined. We know from Analysis (see [2]) that the sum of positive numbers can be defined irrespective of which order we sum the numbers up. However, not all the summands above are necessarily positive. So let us consider the positive and negative terms separately and try to define

EX = ∑_{x∈S_{+}} xp(x) − ∑_{x∈S_{−}}(−x)p(x),

where S_{+} := {x ∈ S : p(x) > 0}, where S_{−} := {x ∈ S : p(x) < 0}. Each of the two sums, separately, is a sum of positive terms, hence it is well-defined. The only “problem” is that such a sum can take value +∞. If both sums are finite then EX is a finite number. If the first sum is +∞ but the second finite then EX = +∞. Similarly, if the first is finite but the second infinite, then EX = −∞. The only case where we cannot talk (cannot define) EX is when both sums are infinite.

When X has absolutely continuous distribution function with density f, one can define EX similarly:

EX = ∫_{0}^{∞} xf(x)dx − ∫_{−∞}^{0}(−x)f(x)dx,

provided that not both integrals are infinity.

In the general case, we need to be more prudent. We will treat the general case, not only because there are random variables which are neither discrete or absolutely continuous but because we can seldom (very seldom indeed) rely on knowledge of the probability mass function or density function.

3.13.1 Definition of expectation

We are going to define the expectation of a random variable X : (Ω, ℱ) → (ℝ, ℬ) with respect to a probability P sitting on (Ω, ℱ). We will use the notation E_{P}X or, simply, EX is the probability P is understood from the context. After constructing this, we will see that it does not depend on the choice of probability space, but only on the law of X, namely, we shall show that if X′ : (Ω′, ℱ′) → (ℝ, ℬ) is another random variable and P′ a probability on (Ω′, ℱ′) such that P_{X} = P′_{X′} then E_{P}X = E_{P′}X′.

▷ SIMPLE RANDOM VARIABLES

The simplest possible random variable is

1_{A}(ω) = \begin{cases}1, & if ω ∈ A, \\ 0, & if ω ∉ A \end{cases}

Since 1_{A} takes value 1 with probability P(A) or 0 with the complementary probability, it is only reasonable to define

E1_{A} = P(A).

We “know” that the expectation should be linear. So we can extend this definition to linear combinations of indicators (simple random variable) and define

E(a_{1}1_{A_{1}} + ⋯ + a_{m}1_{A_{m}}) = a_{1}P(A_{1}) + ⋯ + a_{m}P(A_{m}).  (3.9)

Is this a good definition? There is a slight issue here because a simple random variable can be written as linear combination of indicators in two different ways. For example

81_{A} − 31_{B} = 81_{A\B} − 31_{B\A} + 51_{AB}.

Using the left side we should define the expectation of it 8P(A) − 3P(B), while, using the right side, as 8P(A \ B) − 3P(B \ A) + 5P(AB). So we must have

8P(A) − 3P(B) = 8P(A \ B) − 3P(B \ A) + 5P(AB).

This is true because of additivity of P. Indeed, write P(A) = P(A \ B) + P(AB), P(B) = P(B \ A) + P(AB), substitute in the left side, and you obtain the right side. This argument extends to the general case and so the definition (3.9) is good.

Lemma 3.17 (algebraic properties). Suppose ξ, η are simple random variables on the same probability space (Ω, ℱ , P). Then:

(i) If P(A) = 0 then Eξ1_{A} = 0. (ii) E(cξ) = cE(ξ) for all c ∈ ℝ. (iii) E(ξ + η) = Eξ + Eη. (iv) If P(ξ ≥ 0) = 1 then Eξ ≥ 0. If P(ξ ≤ η) = 1 then Eξ ≤ Eη.

Proof (i) Suppose A is an event in ℱ with P(A) = 0. Then Eξ1_{A} = 0 because: If ξ = ∑_{k} a_{k}1_{A_{k}} then ξ1_{A} = ∑_{k} a_{k}1_{A_{k}A} and so Eξ1_{A} = ∑_{k} a_{k}P(AA_{k}). But P(AA_{k}) ≤ P(A) = 0, for all k. (ii), (iii), (iv) (Almost) immediate from the definition (3.9). □

We now concentrate to nonnegative simple random variables. We allow such a random variable to take value +∞. In other words, ξ is a nonnegative simple random variable if ξ = ∑_{i=1}^{m} a_{i}1_{A_{i}}, with a_{i} ∈ ℝ ∪ {+∞} for all i. The expectation of such a random variable is defined as before: Eξ = ∑_{i=1}^{m} a_{i}P(A_{i}) and the possibility that one of the values may be equal to +∞ worries us not, for all numbers are nonnegative, so the worst that can happen is that Eξ may be +∞.

Lemma 3.18 (analytical properties). Suppose ξ, ξ_{1}, ξ_{2}, …, η_{1}, η_{2}, … are nonnegative simple random variables, and A ∈ ℱ. Then:

(i) If ξ_{n} is an increasing sequence with limit 1_{A} then Eξ_{n} has limit P(A). (ii) If ξ_{n} is an increasing sequence with limit ξ then Eξ_{n} has limit Eξ. (iii) If both ξ_{n}, η_{n} are increasing sequences such that lim_{n} ξ_{n}(ω) = lim_{n} η_{n}(ω) for all ω ∈ Ω, then Eξ_{n}, Eη_{n} converge to the same limit.

Proof (i) Let ε > 0. If ω ∈ A, then ξ_{n}(ω) converges to 1 and so, for all large n, ξ_{n}(ω) > 1 − ε. In other words, the sequence of events {ξ_{n} > 1 − ε} is increasing with union A. Hence P(ξ_{n} > 1 − ε) increases and has limit P(A). But

1_{A} ≥ ξ_{n} ≥ (1 − ε)1(ξ_{n} > 1 − ε).

Therefore, by Lemma 3.17(iv),

P(A) ≥ Eξ_{n} ≥ (1 − ε)P(ξ_{n} > 1 − ε).

Since the last term converges to P(A) and since ε is arbitrary, we have that Eξ_{n} converges to P(A). (ii) Let a_{1}, a_{2}, …, a_{m} be the distinct nonzero values of ξ. Then ξ = ∑_{i=1}^{m} a_{i}1(ξ = a_{i}). Since ξ_{n} ↑ ξ, we have ξ_{n}1(ξ = a_{i}) ↑ ξ1(ξ = a_{i}) = a_{1}1(ξ = a_{i}), and so, a_{i}^{−1}ξ_{n}1(ξ = a_{i}) ↑ 1(ξ = a_{i}). Applying (i), we have E[a_{i}^{−1}ξ_{n}1(ξ = a_{i})] ↑ P (ξ = a_{i}), and so, by Lemma 3.17(ii), E[ξ_{n}1(ξ = a_{i})] ↑ a_{i}P (ξ = a_{i}), for all i = 1, …, m. Summing over i, we have

∑_{i=1}^{m} E[ξ_{n}1(ξ = a_{i})] ↑ ∑_{i=1}^{m} a_{i}P (ξ = a_{i}).

By Lemma 3.17(iii), the left side equals Eξ_{n}, while, by the definition (3.9), the right side equals Eξ. (iii) Since both sequences are increasing and have the same limit we have

lim_{m→∞}(ξ_{n} ∧ η_{m}) = ξ_{n}, lim_{n→∞}(ξ_{n} ∧ η_{m}) = η_{m}.

By (ii), we have

lim_{m→∞} E(ξ_{n} ∧ η_{m}) = Eξ_{n}, lim_{n→∞} E(ξ_{n} ∧ η_{m}) = Eη_{m}.

Since the numerical sequence E(ξ_{n} ∧η_{m}) is increasing in both arguments, we have that the limits over n and over m can be interchanged and so lim_{n→∞} Eξ_{n} = lim_{m→∞} Eη_{m}. □

▷ NONNEGATIVE RANDOM VARIABLES

Suppose that X is a nonnegative random variable that may, possibly, take value +∞. We define

EX := sup{Eξ : ξ is simple nonnegative, and ξ ≤ X}.

We immediately have that there exists a sequence η_{n} of simple random variables such that η_{n} ≤ X for all n and Eη_{n} → EX.

Lemma 3.19. For ANY sequence ξ_{n} of simple nonnegative random variables such that ξ_{n} ↑ X, we have Eξ_{n} ↑ EX.

Proof As we just pointed out, there is one sequence η_{n} of nonnegative simple RVs such that η_{n} ≤ X and Eη_{n} ↑ EX. Let ξ_{n} be a sequence of simple random nonnegative variables such that ξ_{n} ↑ X. Define ζ_{n} = max(η_{1}, …, η_{n}, ξ_{n}). Clearly, ζ_{n} is simple, and ζ_{n} ≤ X. Also, ζ_{n} is increasing with limit X. Since both ξ_{n} and ζ_{n} increase to the same limit, by Lemma 3.18(iii), Eξ_{n} and Eζ_{n} have the same limit. But η_{n} ≤ ζ_{n} ≤ X. Hence Eη_{n} ≤ Eζ_{n} ≤ EX. Since Eη_{n} has limit EX, we conclude that Eζ_{n} has limit EX. Therefore Eξ_{n} has limit EX also. □

In the course of the proof we said that we can find a sequence of simple functions that increases to X. This is done through, for example, the functions τ_{n} defined by (3.1).

τ_{n}(x) := 2^{−n}⌈2^{n}x⌉ ∧ n.

Since

⌈x⌉ = ∑_{k=1}^{∞}1(k − 1 ≤ x < k), x ≥ 0

we can write

τ_{n}(x) := ∑_{k=1}^{n2^{n}} 2^{−n}k1(k − 1 ≤ 2^{n}x < k), x ≥ 0.

Each τ_{n} is left-continuous and τ_{n}(x) → x as n → ∞ for each x. Moreover, τ_{n}(x) ≤ τ_{n+1}(x) for all x. Therefore, for any nonnegative RV X, we have that τ_{n}(X) is an increasing sequence of simple nonnegative RVs with τ_{n}(X) ↑ X and, by Lemma 3.19,

EX = lim_{n→∞} ∑_{k=1}^{n2^{n}} \frac{k}{2^{n}} P(\frac{k − 1}{2^{n}} ≤ X < \frac{k}{2^{n}} ).

Extending Lemma 3.19, we obtain

Theorem 3.5 (monotone convergence theorem). For ANY sequence X_{n} of nonnegative random variables such that X_{n} ↑ X, we have EX_{n} ↑ EX.

Proof By Lemma 3.19, EX = lim_{m→∞} Eτ_{m}(X). By the left-continuity of τ_{m}, we have τ_{m}(X_{n}) ↑ τ_{m}(X), as n → ∞, and by Lemma 3.19, the same is true after taking expectations. So:

EX = lim_{m→∞} lim_{n→∞} Eτ_{m}(X_{n}).

Because the numerical sequence Eτ_{m}(X_{n}) is increasing in both indices m and n, we can interchange the limits:

EX = lim_{n→∞} lim_{m→∞} Eτ_{m}(X_{n}) = lim_{n→∞} EX_{n},

where the last equality follows, again, from Lemma 3.19. □

Lemma 3.20. Let X, Y be nonnegative random variables on (Ω, ℱ , P). Then:

(i) EX = 0 if and only if P(X > 0) = 0. (ii) E(cX) = cEX for all c ≥ 0. (iii) E(X + Y) = EX + EY. (iv) If P(X ≤ Y) = 1 then EX ≤ EY.

Proof We only prove half of (i). The rest are easy. We have {X > 0} is the limit of {X > 1/n}, as n → ∞. Hence P (X > 0) = lim_{n→∞} P (X > 1/n). If P (X > 0) > 0 then P (X > 1/n) > 0 for some n and, since X ≥ (1/n)1(X > 1/n) we have EX ≥ (1/n)P(X > 1/n), i.e. EX > 0. □

Lemma 3.21 (Fatou’s lemma). For any sequence X_{n} of nonnegative random variables, E \underline{lim} X_{n} ≤ \underline{lim} EX_{n}.

Proof We have

inf_{k≥n} X_{k} ≤ X_{ℓ}, ℓ ≥ n.

So, by (iv) of Lemma 3.20,

E inf_{k≥n} X_{k} ≤ EX_{ℓ}, ℓ ≥ n,

and so

E inf_{k≥n} X_{k} ≤ inf_{ℓ≥n} EX_{ℓ}.

Both sides are increasing numerical sequences in n. The right side increases to \underline{lim} EX_{ℓ}. The left side increases, by Theorem 3.5, to E \underline{lim} X_{n}. □

▷ GENERAL RANDOM VARIABLES

Suppose now X is a random variable with values in ℝ. We say that X has an expectation if not both X^{+} = max(X, 0) and X^{−} := −(−X)^{+} have infinite expectations. In this case, we define

EX = EX^{+} − EX^{−}.

The definition is, of course, well-motivated: for any number x, we have x = x^{+} − x^{−}.

EXERCISE 23. If x is a real number, we let, as usual, x^{+} := max(x, 0), x^{−} := −(−x)^{+}. Show that x^{−} = − min(x, 0) and derive the identities

x = x^{+} − x^{−}, |x| = x^{+} + x^{−}, min(|x|, |y|) = \frac{1}{2}(|x| + |y| − |x − y|).

Show also that if a < b then (x ∨ a) ∧ b = (x ∧ b) ∨ a.

We say that X is integrable (with respect to P) if both EX^{+} and EX^{−} are finite or, equivalently, if E|X| < ∞. (The latter follows from the identity |x| = x^{+} + x^{−}.)

Lemma 3.22. Let X, Y be integrable random variables on (Ω, ℱ , P). Then:

(i) E(cX) = cEX for all c ∈ ℝ. (iii) E(X + Y) = EX + EY. (iv) If P(X ≤ Y) = 1 then EX ≤ EY.

If A ∈ ℱ we can define the expectation of X on A by:

E(X; A) := E(X1_{A}).

If P(A) > 0 we can define the expectation of X given A by:

E(X|A) := \frac{E(X; A)}{P(A)}.

We remark that E(X|A) is expectation with respect to the restriction P_{A} of P on A, i.e. with respect to the probability

P_{A} : ℱ → ℝ; P_{A}(B) := P(AB), B ∈ ℱ .

In other words,

E(X|A) = E_{P_{A}}X.

EXERCISE 24. Show that |EX| ≤ E|X| (whenever EX is defined).

Theorem 3.6 (Dominated Convergence Theorem). Let X_{n} be a sequence of random variables such that X(ω) = lim_{n→∞} X_{n}(ω) exists and such that |X_{n}(ω)| ≤ Y (ω) for all n and ω, and E|Y | < ∞. Then E|X_{n} − X| converges to zero.

Proof Apply Fatou’s lemma 3.21 to 2Y − |X_{n} − X|. □

3.13.2 Substitution rule

Lemma 3.23. Consider the measurable functions

(Ω, ℱ) ⟶^{Z} (S, 𝒮) ⟶^{H} (ℝ, ℬ).

Let P be a probability on (Ω, ℱ). Let P_{Z} be the law of Z. Then

E_{P}H^{∘}Z = E_{P_{Z}}H,

whenever wither side exists. (Here, H^{∘}Z is one real random variable on the probability space (Ω, ℱ , P) and H is one real random variable on the probability space (S, 𝒮 , P_{Z}).)

Proof Suppose H is an indicator random variable, i.e. H = 1_{B} for some B ∈ 𝒮. Then E_{P_{Z}}1_{A} = P_{Z}(A) by the definition of the expectation of a simple random variable. On the other hand, H^{∘}Z = 1_{A}(Z) is an indicator random variable on (Ω, ℱ): it is the indicator of the set {ω ∈ Ω : Z(ω) ∈ A}. Hence, again by the by the definition of the expectation of a simple random variable, E_{P}H^{∘}Z = P(Z ∈ A). But P_{Z}(A) = P(Z ∈ A) by the definition of the law of the random variable Z (see section 3.6). Suppose next that H is a simple random variable. Use the above and linearity of expectation to get the result. Suppose that H is a nonnegative random variable. Use Lemma 3.5. Finally, suppose that H has no sign restriction, and use the definition of expectation. □

Corollary 3.5. If X is a real random variable on (Ω, ℱ , P) with expectation E_{P}X and law P_{X} then

E_{P}X = E_{P_{X}} ι

where ι : ℝ → ℝ is the identity function: ι(x) ≡ x.

Therefore, the expectation of a random variable depends only on its law.

Suppose that X is a discrete random variable with values in a finite set S ⊂ ℝ and probability mass function p(x), x ∈ S. Since X = ∑_{x∈S} x1(X = x) is a simple random variable, we immediately have that EX = ∑_{x∈S} xP(X = x) = ∑_{x∈S} xp(x), as needed. The same formula holds for a discrete random variable with values in a countable set S: Simply enumerate the elements of S and use monotone convergence theorem.

Let us now consider an absolutely continuous random variable X with density f. We would like to show that EX is compatible with the definition given at the beginning of the section, namely that it equals ∫_{ℝ} xf(x)dx. To do this, we need to revisit the notion of an integral:

3.13.3 Expectation and densities

Let h : (ℝ, ℬ) → (ℝ, ℬ) be measurable and suppose h ≥ 0. Consider the random variable U on the coin-flip space (Ω = {0, 1}^{ℕ}, ℱ , P), defined by (3.6). We define ∫_{ℝ} h(x)dx by

∫_{ℝ} h(x)dx := ∑_{n∈ℤ} Eh(U + n).

This is called the Lebesgue integral of h. We also define ∫_{a}^{b} h(x)dx by

∫_{a}^{b} h(x)dx := ∫_{ℝ} h(x)1(a ≤ x ≤ b)dx,

and, more generally, for any B ∈ ℬ,

∫_{B} h(x)dx := ∫_{ℝ} h(x)1(x ∈ B)dx.

If h has no restriction on sign, we define ∫_{a}^{b} h(x)dx as ∫_{a}^{b} h^{+}(x)dx − ∫_{a}^{b} h^{−}(x)dx provided that not both terms are infinite. This integral behaves like the ordinary Riemann integral for “nice” functions (e.g. for continuous or piecewise continuous functions).

More generally, if F : ℝ → ℝ is an increasing function, we define

∫_{ℝ} h(x)F (dx) = ∫_{ℝ} h(F^{−1}(x))dx,

where F^{−1}(x) := inf{t ∈ ℝ : F (t) > x}. This is the Lebesgue-Stieltjes integral. It is also denoted by ∫_{ℝ} h(x)dF (x) or simply as ∫_{ℝ} hdF when the variable of integration needs no mentioning.

The following theorem connects what one learns in basic Calculus with what we just defined. You absolutely need this theorem in order to make the connection between the standard Calculus tricks and recipes^{g} and what we are talking about here.

Theorem 3.7. (i) If h is Riemann integrable on [a, b] then the Lebesgue integral ∫_{a}^{b} h(x)dx exists and coincides with the Riemann integral. (ii) If h is bounded and measurable then it is Riemann integrable on [a, b] if the set of discontinuities of h have measure zero, in the sense of Definition 3.2.

Example 3.8. For example, if h is piecewise continuous, then its discontinuities have measure zero and so it is Riemann integrable.

Example 3.9. As a counterexample, consider the function h(x) = 1(x ∈ ℚ), 0 ≤ x ≤ 1. This function is discontinuous on every point of [0, 1]. The interval [0, 1] does not have measure zero. Hence it is not Riemann integrable. However, h is measurable and nonnegative, and so it does have a Lebesgue integral: this integral equals Eh(U) = P(U ∈ ℚ) = 0.

Theorem 3.7 enables us to use all rules we learn in basic Integration Calculus to the case of Lebesgue integration when the two integrals coincide.

Suppose now that X is a random variable with absolutely continuous distribution function F and density f. Consider the function Q : ℬ → ℝ defined by

Q(B) := ∫_{B} f(x)dx.

Since Q(B) = P_{X}(B) for B = (−∞, x], we have that Q = P_{X}. More generally, Lemma 3.24. Let P_{X} be the law of a random variable X with density f. Then, for any measurable g : (ℝ, ℬ) → (ℝ, ℬ),

gI mean things like ∫_{0}^{x} t^{2}e^{t}dt = x^{2}e^{x} − 2 ∫_{0}^{x} te^{t}dt = (x^{2} − 2x + 2)e^{x} − 2.

∫_{ℝ} g(x)f(x)dx = E_{P_{X}} g

provided that wither side exists.

Proof If g = 1_{B}, then this is what was discussed before the Lemma. For g simple, we use linearity. For general g we approximate. □

Corollary 3.6. Let P_{X} be the law of an integrable random variable X with density f.

(i) If ι is the identity function on ℝ then

∫_{ℝ} xf(x)dx = E_{P_{X}} ι.

(ii)

EX = ∫_{ℝ} xf(x)dx.

Proof (i) follows from Lemma 3.24 with g = ι. (ii) follows from corollary 3.5 □

EXERCISE 25. Suppose that Z is a real random variable with absolutely continuous distribution function and density f_{Z}. Let H : (ℝ, ℬ) → (ℝ, ℬ) be a measurable function. Suppose that the random variable X = H(Z) has density f_{X}. Show that the expectation of X (if it exists) can be computed in two ways:

EX = ∫_{ℝ} xf_{X}(x)dx = ∫_{ℝ} H(z)f_{Z}(z)dz.

EXERCISE 26. Let B ∈ ℬ and let λ(B) := ∫_{ℝ} 1_{B}(x)dx (Lebesgue integral). Show that λ : ℬ → ℝ satisfies λ(∪_{n}A_{n}) = ∑_{n} λ(A_{n}) whenever the A_{n} are mutually disjoint elements of ℬ and that λ(A + t) = λ(A) for all A ∈ ℬ, t ∈ ℝ, where A + t := {a + t : a ∈ A}. The function λ is called length.

EXERCISE 27. Consider a compass with a laser pointer attached at both ends of the needle. Suppose there is an infinite screen at some distance from the compass. Give it a spin and see mark X the location of the light with respect to a fixed point O on the screen (positive if it is to the right of O; negative if it is to the left). Show that EX is not defined. (You first must translate this problem in Mathematics.)

EXERCISE 28. Consider the function F (x) := ∑_{k∈ℕ} 1(x ≤ k), x ≥ 0, F (x) = 0, x < 0. Show that

∫_{ℝ} h(x)F (dx) = ∑_{k=1}^{∞} h(k),

whenever the sum on the right makes sense.

3.14 Inequalities

Mathematics needs inequalities probably more than equalities. Probability, in particular, which, in some sense, contains a lot of approximation ideas needs inequalities. This section discusses some basic ones.

SMST C: Probability 3–29

3.14.1 Markov, Chebyshev, Chernoff

Lemma 3.25 (Markov inequality). If X is a nonnegative random variable then

P(X ≥ t) ≤ \frac{EX}{t} , t > 0.

Proof We have

t1(X ≥ t) ≤ X

and E is preserved by ≤. □ Definition 3.4. The variance of a real random variable X with EX^{2} < ∞ is defined by

var X := E(X − EX)^{2}.

Lemma 3.26 (Chebyshev inequality). If X is a real random variable with EX^{2} < ∞ then

P(|X − EX| ≥ t) ≤ \frac{var X}{t}

Proof Apply the Markov inequality to |X − EX|. □ Lemma 3.27 (Chernoff inequality). If X is a real random variable then

P(X ≥ t) ≤ \frac{Eg(X)}{g(t)}

where g is a positive increasing function.

Proof Since g is increasing,

{X ≥ t} ⊂ {g(X) ≥ g(t)}

Now apply the Markov inequality to g(X). □ EXERCISE 29. Let X be a discrete random variable with P(X = k) = \binom{n}{k}2^{−k}, k = 0, 1, …, n. Estimate P(X > na) for a > 0.5 using the above inequalities.

3.14.2 Jensen

A function φ : ℝ → ℝ is convex if

φ(pa + (1 − p)b) ≤ pφ(a) + (1 − p)φ(b)

for all a, b ∈ ℝ and all 0 ≤ p ≤ 1. Notice that if ξ is a random variable with P(ξ = a) = p, P(ξ = b) = 1 − p, this definition can be written as

φ(Eξ) ≤ Eφ(ξ).

Jensen’s inequality generalises this observation: Lemma 3.28. Let X be a real integrable random variable and φ a convex function. Then

φ(EX) ≤ Eφ(X).

Geometrically, a convex function is such that the graph of the function on the interval [a, b] lies below the straight segment with endpoints (a, φ(a)), (b, φ(b)).

If we take a straight line with slope equal to the slope of this segment and move it down, at some point it will be entirely below the whole graph of φ. Moreover, if we consider the set of all straight lines that are below the graph of φ and take their envelope (the maximum) then this equals φ. Namely,

φ(x) = max_{ℓ∈L(φ)} ℓ(x), x ∈ ℝ,  (3.10)

where L(φ) is the collection of all functions ℓ(x) ≡ αx + β such that ℓ ≤ φ. To show this rigorously, observe that

a < b < c ⇒ \frac{φ(b) − φ(a)}{b − a} ≤ \frac{φ(c) − φ(b)}{c − b},

by the definition of convexity. Let ε > 0, a = b − ε, c = b + ε and so

D_{−}φ(b) := lim_{ε↓0} \frac{φ(b) − φ(b − ε)}{ε} ≤ lim_{ε↓0} \frac{φ(b + ε) − φ(b)}{ε} := D_{+}φ(b),

where the limits exist by monotonicity. Let α be between D_{−}φ(b) and D_{+}φ(b). Then, if x > b+ε, we have

\frac{φ(b + ε) − φ(b)}{ε} ≤ \frac{φ(x) − φ(b)}{x − b},

and so

α ≤ D_{+}φ(b) ≤ \frac{φ(x) − φ(b)}{x − b}, x > b,

or α(x − b) + φ(b) ≤ φ(x), for all x ≥ b. Arguing similarly for all x ≤ b, we have

α(x − b) + φ(b) ≤ φ(x), x ∈ ℝ.

Thus, the function ℓ(x) := α(x − b) + φ(b) ≤ φ(x) belongs to L(φ) and also, ℓ(b) = φ(b). This proves (3.10). Proof of Lemma 3.28:

φ(EX) = max_{ℓ∈L(φ)} ℓ(EX) = max_{ℓ∈L(φ)} Eℓ(X) ≤ E max_{ℓ∈L(φ)} ℓ(X) = Eφ(X).

The first equality is due to (3.10). The second uses linearity of expectation. The third uses monotonicity of expectation. The last reuses (3.10). □

EXERCISE 30. Let a_{1}, …, a_{n} be positive real numbers. Define their arithmetic, geometric and harmonic mean by

A_{n} = \frac{a_{1} + ⋯ + a_{n}}{n} , G_{n} = (a_{1} ⋯ a_{n})^{1/n}, H_{n} = \frac{n}{a_{1}^{−1} + ⋯ + a_{n}^{−1}} ,

respectively, and show that A_{n} ≥ G_{n} ≥ H_{n}.

3.15 Moments

Definition 3.5. When r > 0, the r-moment of a nonnegative RV X is defined as the quantity EX^{r}. The r-norm of a real RV X is defined as ||X||_{r} := (E|X|^{r})^{1/r}.

Lemma 3.29. The r-norm of X is increasing in r.

Proof Let r < s and φ(x) = x^{s/r}, x > 0. Notice that φ is convex. (This follows from the fact that its second derivative is positive.) Now apply the Jensen inequality. □

Corollary 3.7. If E|X|^{p} < ∞ for some p > 0 then E|X|^{r} < ∞ for all 0 < r < p.

3.16 Hölder, Minkowski and Cauchy-Bunyakowskii-Schwarz

Definition 3.6. If X, Y are real random variables on the same (Ω, ℱ , P), the quantity E(XY) (whenever it is defined) is called correlation between X and Y. The quantity cov(X, Y) := E((X − EX)(Y − EY)) is called covariance between X and Y.

Lemma 3.30 (Hölder inequality). Let X, Y be real random variables. Then

|E(XY)| ≤ ||X||_{p}||Y ||_{q},

for any p, q > 0, p^{−1} + q^{−1} = 1, as long as all terms involved exist and are finite.

Proof Let (Ω, ℱ , P) be a probability space on which both X, Y are defined. Without loss of generality assume that they are both nonnegative. Let q > 1 and assume that E(Y^{q}) < ∞. Consider the probability

P_{q}(A) := \frac{E(Y^{q}1_{A})}{E(Y^{q})}, A ∈ ℱ .

Let E_{q} denote expectation with respect to P_{q}. Therefore, for any nonnegative random variable W : (Ω, ℱ) → (ℝ, ℬ),

E_{q}W = \frac{E(Y^{q}W)}{E(Y^{q})}.  (3.11)

Letting, in (3.11), W = XY^{1−q}, we obtain

E(XY) = E(Y^{q}) E_{q}(XY^{1−q}).

Let p be defined from p^{−1} + q^{−1} = 1. Necessarily, p > 1. From Lemma 3.29 we have

E_{q}Z ≤ (E_{q}Z^{p})^{1/p},

for any nonnegative random variable Z : (Ω, ℱ) → (ℝ, ℬ) with EZ^{p} < ∞. Therefore,

E(XY) ≤ E(Y^{q}) (E_{q}((XY^{1−q})^{p}))^{1/p}

= E(Y^{q}) (\frac{E(X^{p}Y^{q}Y^{(1−q)p})}{E(Y^{q})})^{1/p}

= (E(Y^{q}))^{1−1/p} (E(X^{p}Y^{q+(1−q)p}))^{1/p}.

Since 1 − 1/p = 1/q and q + (1 − q)p = 0, the result follows. □

Corollary 3.8. (Cauchy-Bunyakowskii-Schwarz) Let X, Y be real random variables. Then

|E(XY)| ≤ ||X||_{2}||Y ||_{2},

as long as all terms involved exist and are finite.

Proof Notice that \frac{1}{2} + \frac{1}{2} = 1 and apply Hölder. □

Corollary 3.9. Let X, Y be real random variables. Let

ρ(X, Y) := cov(X, Y)/\sqrt{var(X)}\sqrt{var(Y)},

whenever the terms exist. Then

−1 ≤ ρ(X, Y) ≤ 1.

Lemma 3.31 (Minkowski inequality). Let X, Y be real random variables. Then

||X + Y ||_{p} ≤ ||X||_{p} + ||Y ||_{p},

for any p > 1, as long as all terms involved exist and are finite.

Proof Use the Hölder inequality as follows:

E(|X + Y |^{p}) = E(|X| |X + Y |^{p−1}) + E(|Y | |X + Y |^{p−1})
≤ [E(|X|^{p})]^{1/p} [E(|X + Y |^{(p−1)q})]^{1/q} + [E(|Y |^{p})]^{1/p} [E(|X + Y |^{(p−1)q})]^{1/q}
= (||X||_{p} + ||Y ||_{p}) [E(|X + Y |^{p})]^{1/q},
□

3.17 Moment generating functions

Let X be a real random variable. Since, for any θ ∈ ℝ, the random variable e^{θX} is nonnegative, its expectation exists (but may be equal to +∞). We define the function M : ℝ → ℝ ∪ {+∞} by

M(θ) := E(e^{θX}), θ ∈ ℝ.

Notice that M(0) = 1. This function is useful if M(θ) < ∞ for some θ ≠ 0. (Indeed, there are cases where θ = 0 is the only point at which M is finite.) If X is a positive random variable, then M(θ) < ∞ for all θ ≤ 0. If X is a negative random variable, then M(θ) < ∞ for all θ ≥ 0. M depends only on the law of X. Indeed, using the Substitution Rule (Lemma 3.23) we can write

M(θ) = E_{P_{X}} (e^{θι}), where ι(x) ≡ x,

and, if X has absolutely continuous distribution function F with density f, we can write

M(θ) = ∫_{ℝ} e^{θx}f(x)dx (Lebesgue integral) .

The function M is called moment generating function because of the following:

Lemma 3.32. Suppose there exist a < 0 < b such that M(θ) < ∞ for all a < θ < b. Then

(i) the r-moment of X exists for all r ∈ ℕ and is given by the r-derivative of M at 0:

E(X^{r}) = D^{r}M(0).

(ii)

M(θ) = ∑_{r=0}^{∞} \frac{E(X^{r})}{r!}θ^{r}, a < θ < b.

(iii) There is only one distribution function F such that if X has distribution function F then it has moment generating function M.

Proof [sketch] Using the Dominated Convergence Theorem 3.6, we can see that M is infinitely differentiable at 0 with r-derivative equal to the r-moment of X. Moreover, we can see that M is a real analytic function around 0. Hence Taylor’s theorem holds, which yields the second claim. □

▷ ▷ ▷ ▷ ▷ APPENDICES ◁ ◁ ◁ ◁ ◁

3.18 APPENDIX A: Sets

The logic of events plays a much more fundamental rôle in the whole subject than the layman could imagine. We review basic concepts here.^{h}

§ We express the statement that x is an element of (or belongs to) a set A by x ∈ A. The negation of the latter is denoted by x ∉ A. We write A ⊂ B for to express the statement “x ∈ A ⇒ x ∈ B”. We let A∩B be the intersection and A∪B the union of A and B. To save space, we often write AB in lieu of A∩B and ABC in lieu of A∩B∩C. The set A^{c} is defined through: “x ∈ A^{c} ⟺ x ∉ A”. Naïve set theory and first order logic are algebraically equivalent through the above and through: “A ∩ B ⟺ x ∈ A and x ∈ B”, “A ∪ B ⟺ x ∈ A or x ∈ B”. We let B \ A = B ∩ A^{c} and we write B − A for B \ A if A ⊂ B. The symmetric difference (‘exclusive or’) A△B is defined as the set of elements of A or B which do not belong to both, i.e. A△B = (A \ B) ∪ (B \ A) = (A ∪ B) − (A ∩ B). If {A_{j}, j ∈ J} is a collection of sets we let ∩_{j∈J}A_{j} be their intersection and ∪_{j∈J}A_{j} their union. Here, J is an arbitrary set. If J = ℕ we write ∩_{j∈ℕ}A_{j} = ∪_{j=1}^{∞}A_{j}, and, as usually in Mathematics, the convention is that the symbol ∞ appearing on top is not part of the index set (∞ is not a natural number).^{i} Here is an example of how we translate elementary logic (i.e. ordinary language, say English) in Mathematics: Consider the sentence

S = [after some day it will never rain again in Glasgow].

If we let R_{i} = [it rains on the i-th day in Glasgow]

we can write S as

S = [∃i ∀j ≥ i notR_{i}].

Taking the negation of this sentence^{j} we get:

notS = not[∃i ∀j ≥ i notR_{i}] = [∀i ∃j ≥ i R_{i}].

Translating this back into English, it reads:

notS = [for all days there will be a day in the future during which it rains in Glasgow]

and, since this is not too palatable even for the literati, we can express it, equivalently, as

notS = [it rains in Glasgow infinitely often].

What does this have to do with sets? Well, imagine there is a big set (the “universe”)–call it Ω–containing all possible states of Glasgow at all days. At the minimum, we want to know whether it rains or not on each day. So we decide to let an element of Ω be the sequence (ω_{1}, ω_{2}, …) of states of Glasgow, where ω_{i} takes values “rain” or “doesntrain” and represents what happens on the i-th day in Glasgow. So Ω is the set of all these sequences, mathematically written as Ω = {rain, doesntrain}^{ℕ}. Then R_{i} can be taken to be the set of all sequences ω ∈ Ω such that ω_{i} = rain. You can then check that the set (subset of Ω) S = ∪_{i} ∩_{j≥i} R_{i}^{c} corresponds to the statement S, while S^{c} = ∩_{i} ∪_{j≥i} R_{i} corresponds to the statement notS. In this sense, the so-called Kolmogorov model of Probability merely puts within a mathematical framework (that of Set Theory) what we intuitively feel as assigning numbers representing chance to statements, to events, to situations describable by ordinary language. So Probability Theory is ordinary logic together with numbers assigned to statements. We talk about logic in this Appendix and about numbers in the next.

hThere are no numbers in this section; only sets. (Of course, numbers are sets; so the above declaration is offered with apologies to set theorists.)

iCompare this with a sum of real numbers: ∑_{j=1}^{∞} x_{j} means that j runs over 1, 2, 3 … but does not take the value ∞.

j To which, I think, we can safely assign the truth value

§ If A is a set then 2^{A} is the set of its subsets. A set of sets (or “collection” of sets) is a subset of 2^{A}. If A has n elements then 2^{A} has 2^{n} elements. If ℕ is the set of natural numbers then 2^{ℕ} has as many elements as the set of real numbers. (And why, on earth, should we use the symbol 2^{A}? To explain, consider a simple set, say A = {x, y, z}. I can form a subset of A by putting a 1 to each element I pick or 0 to each element I do not pick. For instance, 011 means “do not pick x, pick y, pick z”, i.e. it refers to the subset {y, z}. We thus see there is a correspondence between subsets of A and triples of 0’s and 1’s. How many such triples do we have? Obviously, 2 × 2 × 2 = 2^{3}. This explains why we have 2^{3} = 8 subsets. If A had n elements, you see there are 2^{n} subsets and so the notation 2^{A} is suggestive of this, and we carry it on even to cases where A has infinite number of elements.)

§ If f : X → Y is a function then, for any B ⊂ Y we define f^{−1}(B) = {x ∈ X : f(x) ∈ B}, and for any A ⊂ X we define f(A) = {f(x) : x ∈ A}. Thus f can be lifted as a map from 2^{X} into 2^{Y} and we also have a map f^{−1} from 2^{Y} into 2^{X}. We always have f(f^{−1}(B)) ⊂ B and f^{−1}(f(A)) ⊃ A. Also, f^{−1}(∩_{j}B_{j}) = ∩_{j}f^{−1}(B_{j}) f^{−1}(∩_{j}B_{j}) = ∩_{j}f^{−1}(B_{j}), f^{−1}(B^{c}) = f^{−1}(B)^{c}. However, f as a map on sets does not behave that well (fortunately!) because f(∩A_{j}) ⊂ ∩_{j}f(A_{j}) (equality, in general, fails), but f(∪A_{j}) = ∪_{j}f(A_{j}); also, f(A_{2} − A_{1}) ⊂ f(A_{2}) − f(A_{1}).

§ The set of functions from X into Y is denoted by Y^{X}. The notation is motivated by the fact that the set of functions from {1, …, n} into ℝ is “the same” as ℝ^{n}. Consider, for example, the set of functions from a set A into {0, 1}. This is denoted as {0, 1}^{A}. Since there is a one-to-one correspondence between this set of functions and the collection of subsets 2^{A} of A (as explained above), the surprise that you may have just experienced with this new notation should not be as high.

§ If A ⊂ Ω the function 1_{A} : Ω → ℝ, defined by 1_{A}(ω) = 1 if ω ∈ A, and 0, otherwise, is the indicator^{k} of the set A. Note that 1_{A}1_{B} = 1_{AB}, 1_{A^{c}} = 1 − 1_{A}. If A ⊂ B, then 1_{B−A} = 1_{B} − 1_{A}. If AB = ∅ then 1_{A∪B} = 1_{A} + 1_{B}. We can check that the ‘inclusion-exclusion formula’ 1_{A∪B} = 1_{A} + 1_{B} − 1_{AB} holds for arbitrary A, B, by (if we want to be slick) using (1 −1_{A})(1 −1_{B}) = 1 −1_{A} −1_{B} +1_{A}1_{B} and using the fact that 1_{A}1_{B} = 1_{AB} while (1 −1_{A})(1 −1_{B}) = 1_{A^{c}}1_{B^{c}} = 1_{A^{c}B^{c}} = 1(A ∪ B)^{c} = 1 −1_{A∪B}. The inclusion-exclusion formula for many sets follows by expanding the product (1 − 1_{A_{1}}) ⋯ (1 − 1_{A_{n}}).

§ Often, sets have elements which are themselves sets. (For example, the set of all sets of dishes in a shop.) When we talk about sets of sets we refer to them as classes (or collections, or systems, or aggregates) of sets.^{l}

§ A class of sets ℛ ⊂ 2^{Ω} is a π-system if it is closed under pairwise intersections (A, B ∈ ℛ ⇒ AB ∈ ℛ). If 𝒜 ⊂ 2^{Ω} then the intersection of all π-systems containing 𝒜 is a π-system and is denoted by π(𝒜).

§ A class of sets 𝒟 ⊂ 2^{Ω} is a λ-system if it is closed under increasing differences (A, B ∈ 𝒟, A ⊂ B ⇒ AB ∈ 𝒟), and increasing limits (A_{n} ∈ 𝒟, A_{n} ⊂ A_{n+1}, for all n, ⇒ ∪_{n} ∈ 𝒟). If 𝒜 ⊂ 2^{Ω} then the intersection of all λ-systems containing 𝒜 is a λ-system and is denoted by λ(𝒜).

§ A class of sets 𝒞 ⊂ 2^{Ω} is a field (or algebra) if it is a π system containing Ω and such that A ∈ 𝒞 ⇒ A^{c} ∈ 𝒞. (Therefore, a field is closed under unions and any finite number of operations on its elements.) If 𝒜 ⊂ 2^{Ω} then the intersection of all fields containing 𝒜 is a field.

§ A class of sets ℱ ⊂ 2^{Ω} is a σ-field (or σ-algebra)^{m} if it is a field and if A_{1}, A_{2}, … ∈ ℱ ⇒ ∩_{j=1}^{∞}A_{j} ∈ ℱ. If 𝒜 ⊂ 2^{Ω} then the intersection of all λ-systems containing 𝒜 is a σ-field and is denoted by σ(𝒜).

kAlso called characteristic function, but, mostly, in Analysis. In Probability, if 1_{A} is thought of as a random variable, it is also called Bernoulli random variable.

lFor philological, rather than mathematical, reasons

mThe letter σ is supposed to mean “sum” and, in fact, a “sum” of countably many objects; in the case at hand it represents “countable unions”

§ If {ℱ_{j}} is a collection of σ-fields on the same set then ∩_{j}ℱ_{j} is a σ-field too: it is the largest σ-field contained in all of them. (The intersection of σ-fields is never empty because every σ-field contains the empty set.) However the union of σ-fields is not a nice object and we don’t consider it. Instead, we define ∨_{j}ℱ_{j} to be the smallest σ-field containing all the ℱ_{j}.

§ If A, B are sets, then A × B is the set of all ordered pairs (a, b) where a ∈ A and b ∈ B. Similarly, if {A_{j}, j ∈ ℕ} is a sequence of sets then ×_{j=1}^{∞}A_{j} is the set of all sequences (a_{1}, a_{2}, …) where a_{j} ∈ A_{j} for all j. Finally, if {A_{t}, t ∈ T } is an arbitrary collection of sets we may defined ×_{t∈T} A_{t} as the collection of functions f ∈ A^{T}, where A = ∪_{t∈T} A_{t} such that f(t) ∈ A_{t} for all t ∈ T. Notice that this definition does not respect any “order” in the index set T even if this is there; the order imposed in the definition of the Cartesian product of finitely or countably many sets is an extra bag that we have been forced to carry (for good reasons). When A = ×_{j∈J}A_{j} is a Cartesian product, we can define, for each i ∈ J, the projection function ρ_{i} : A → A_{i} by ρ_{i} : (a_{j}, j ∈ J) ↦ a_{i}.

§ Let (Ω_{1}, ℱ_{1}), (Ω_{2}, ℱ_{2}) be measurable spaces. Then ℱ_{1} ⊗ ℱ_{2} is the σ-field on Ω_{1} × Ω_{2} generated by sets of the form A_{1} × A_{2}, where A_{1} ∈ ℱ_{1}, A_{2} ∈ ℱ_{2}. It also equals σ(ρ_{1}, ρ_{2}), where ρ_{1}, ρ_{2} are the projection functions. It also equals the smallest σ-field containing the σ-field {A_{1} × Ω_{2} : A_{1} ∈ ℱ_{1}} and the σ-field {Ω_{2} × A_{2} : A_{2} ∈ ℱ_{2}}. The definition is extended for any finite product.

§ The Borel σ-field ℬ on ℝ is the σ-field generated by the class of open sets. (A set O is open if for any any x ∈ O there is an ε > 0 such that [x − ε, x + ε] ∈ O.) It is also generated by intervals of the form (a, b), a, b ∈ ℝ. It is also generated by intervals of the form [a, b], a, b ∈ ℝ. It is also generated by intervals of the form (−∞, x], x ∈ ℝ. (This last class of intervals forms a π-system.) The Borel σ-field ℬ(ℝ^{2}) on ℝ^{2} = ℝ × ℝ is defined by ℬ(ℝ^{2}) = ℬ ⊗ ℬ. Similarly, we define ℬ(ℝ^{d}) for any natural number d, as ℬ(ℝ^{d}) = ℬ(ℝ^{d−1}) ⊗ ℬ, by induction on d. Notice that ℬ(ℝ^{d}) is also generated by open sets in ℝ^{d}. (A set O ⊂ ℝ^{d} is open if for any x = (x_{1}, …, x_{d}) ∈ ℝ^{d} there is an ε > 0 such that the cube [x_{1} − ε, x_{1} + ε] × ⋯ × [x_{d} − ε, x_{d} + ε] ∈ O.)

§ Let (S_{i}, 𝒮_{i}), i ∈ ℕ be measurable spaces, and let S = ×_{i}S_{i}. A rectangle is a subset of S of the form B_{1} × ⋯ × B_{j} × ⋯ with B_{i} ∈ 𝒮_{i} for all i. A finite-dimensional rectangle is a rectangle such that B_{k} = S_{k} for all k greater than some index j. A cylinder set is a subset of S of the form A × S_{j+1} × S_{j+2} × ⋯, where A ∈ 𝒮_{1} ⊗ ⋯ ⊗ 𝒮_{j}. A finite-dimensional rectangle is a rectangle and a cylinder set. A rectangle is not necessarily a cylinder set. The collection of rectangles is π-system (the intersection of two rectangles is still a rectangle; but the union of two rectangles is not a rectangle; the complement of a rectangle is not a rectangle). Ditto for finite-dimensional rectangles. The collection of cylinder sets is a field (the intersection of two cylinder sets is a cylinder set and the complement of a cylinder set is a cylinder set), but not a σ-field. On S we can put a natural σ-field, denoted by ⨂_{i} 𝒮_{i}: it is the σ-field generated by cylinder sets; it is also generated by rectangles; it is also generated by finite-dimensional rectangles; it is also the smallest σ-field such that every projection π_{i} : (S, 𝒮) → (S_{i}, 𝒮_{i}) is a random variable.

§ Let A_{n} be a sequence of subsets of Ω. We define^{n}

\overline{lim}_{n→∞} A_{n} = ∩_{n} ∪_{m≥n} A_{m}, \underline{lim}_{n→∞} A_{n} = ∪_{n} ∩_{m≥n} A_{m},

and notice that (\overline{lim}_{n→∞} A_{n})^{c} = \underline{lim}_{n→∞} A_{n}^{c}. We say that lim_{n→∞} A_{n} = A (we also write A_{n} → A) if, by definition, \overline{lim}_{n→∞} A_{n} = \underline{lim}_{n→∞} A_{n} = A, and write this also as A_{n} → A. For example, if A_{n} is an increasing sequence of sets then (show this!) lim_{n→∞} A_{n} = ∪_{n}A_{n} and if A_{n} is decreasing then lim_{n→∞} A_{n} = ∩_{n}A_{n}. Also notice that ω ∈ \overline{lim} A_{n} if and only if ω belongs to A_{n} for infinitely many indices n, while ω ∈ \underline{lim} A_{n} if and only if ω belongs to A_{n} for all except finitely many n.

nNotice the complete analogy with numerical sequences: \overline{lim}_{n→infty} x_{n} = inf_{n} sup_{k≥n} x_{k}, \underline{lim}_{n→infty} x_{n} = sup_{n} inf_{k≥n} x_{k}; and x_{n} → x, as n → ∞, if and only if \overline{lim} x_{n} = \underline{lim} x_{n} = x; the limit x could be ±∞.

§ If 𝒮 is a λ-system and A ⊂ Ω then

𝒮_{A} := {B ∈ 2^{Ω} : BA ∈ 𝒮 }

is also a λ-system. Indeed, if B_{1} ⊂ B_{2} both belong to 𝒮_{A} then B_{1}A, B_{2}A belong to 𝒮 and (B_{2} −B_{1})A = B_{2}A − B_{1}A ∈ 𝒮, because 𝒮 is closed under proper differences; so B_{2} − B_{1} ∈ 𝒮_{A}. Also, if B_{n} ∈ 𝒮_{A}, B_{n} ⊂ B_{n+1} for all n, then B_{n}A ∈ 𝒮 for all n, and so A ∪_{n} B_{n} = ∪_{n}AB_{n} ∈ 𝒮 because 𝒮 is closed under increasing limits; so sup_{n} B_{n} ∈ 𝒮_{A}.

§ π & λ & Ω ≡ σ. In other words, if 𝒮 is both π and λ and contains Ω then it is a σ-field. Indeed, if A ∈ 𝒮 then A^{c} = Ω − A ∈ 𝒮 because 𝒮 is λ. So 𝒮 is closed under complementations, intersections and limits. So it is a σ-field.

§[Sierpiński-Dynkin Lemma] Let 𝒮 be a π-system containing Ω. Then λ(𝒮) = σ(𝒮). Proof Since λ(𝒮) is the smallest λ-system containing 𝒮 and since σ(𝒮) is a λ-system, we have λ(𝒮) ⊂ σ(𝒮). We need to show that σ(𝒮) ⊂ λ(𝒮). If we show that λ(𝒮) is also a π-system then it will be a σ-field and we’ll be done. We know that

A ∈ 𝒮 , B ∈ 𝒮 ⇒ AB ∈ 𝒮 ⊂ λ(𝒮).  (3.12)

Hence

∀A ∈ 𝒮 𝒮 ⊂ {B ∈ 2^{Ω} : AB ∈ λ(𝒮)} =: λ(𝒮)_{A}.

But λ(𝒮)_{A} is λ and so

∀A ∈ 𝒮 λ(𝒮) ⊂ λ(𝒮)_{A}.

Let’s read this again:

A ∈ 𝒮 , B ∈ λ(𝒮) ⇒ AB ∈ λ(𝒮).  (3.13)

Wonder of wonders: (3.12) implied the seemingly stronger (3.13). Let’s rewrite the (3.13) as:

∀B ∈ λ(𝒮) 𝒮 ⊂ {A ∈ 2^{Ω} : AB ∈ λ(𝒮)} =: λ(𝒮)_{B}.

But λ(𝒮)_{B} is λ and so

∀B ∈ λ(𝒮) λ(𝒮) ⊂ λ(𝒮)_{B}.

Let’s read this again:

B ∈ λ(𝒮), A ∈ λ(𝒮) ⇒ AB ∈ λ(𝒮).  (3.14)

Wonder of wonders of wonders: (3.13) implied the seemingly stronger (3.14) which is what we need.

3.19 APPENDIX B: Sets and numbers

In some sense, probability is all about assigning numbers to sets.

§ Let Ω be a set and 𝒞 a field of subsets of it. The function P_{0} : 𝒞 → [0, 1] is additive if P_{0}(A ∪ B) = P_{0}(A) + P_{0}(B) whenever AB = ∅. It is countably additive if for any A ∈ 𝒞 for which there exist mutually disjoint A_{n} ∈ 𝒞 with A = ∪_{n}A_{n} we have P_{0}(A) = ∑_{n} P_{0}(A_{n}). We will throughout assume that P_{0}(Ω) = 1. (If, in addition to the above, we have that 𝒞 is a σ-field, then P_{0} is a probability.)

§ If P is a probability on (Ω, ℱ), then (Ω, ℱ , P) is called probability space. If A_{n} is a sequence of sets in ℱ then P(∪_{n}A_{n}) ≤ ∑_{n} P(A_{n}) [Boole’s inequality]. If A_{n} → A as n → ∞ then P(A_{n}) → P(A) as n → ∞ [continuity]. If ∑_{n} P(A_{n}) < ∞ then P(\overline{lim} A_{n}) = 0. [Borel-Cantelli lemma]. (Proof: ∪_{k≥m}A_{k} → \overline{lim} A_{n}, as m → ∞; So P(\overline{lim}_{n} A_{n}) = lim_{k≥m} P(∪_{k≥m}A_{k}). But the latter is ≤ ∑_{k≥m} P(A_{k}) which tends to 0, as m → ∞, by the assumption.)

Lemma [CA] The following are equivalent:

(i) P_{0} is countably additive on the field 𝒞. (ii) P_{0}(A_{n}) → 0 for all decreasing sequences A_{n} ∈ 𝒞 with ∩_{n}A_{n} = ∅. (iii) P_{0}(A_{n}) → 0 for all sequences A_{n} ∈ 𝒞 with A_{n} → ∅. Proof Suppose (i) holds. Let A_{n} decrease to ∅. We have

P_{0}(A_{1}) = ∑_{k=1}^{∞} P_{0}(A_{k} − A_{k+1})

= lim_{n→∞} ∑_{k=1}^{n−1}[P_{0}(A_{k}) − P_{0}(A_{k+1})]

= lim_{n→∞}[P_{0}(A_{1}) − P_{0}(A_{n})],

i.e. P_{0}(A_{n}) → 0. We proved (i) ⇒ (ii). Next, suppose (ii) holds. Let A_{n} be mutually disjoint elements of 𝒞 with ∪_{n}A_{n} ∈ 𝒞. By finite additivity,

P_{0}(∪_{k}A_{k}) = ∑_{k=1}^{n} P_{0}(A_{k}) + P_{0}(∪_{k>n}A_{k}).

Since ∪_{k>n}A_{k} decreases to ∅ we have P_{0}(∪_{k>n}A_{k}) → 0, and so we obtain countable additivity. We proved (ii) ⇒ (i). Suppose (ii) holds. Let A_{n} ∈ 𝒞, A_{n} → ∅. We want to show that P_{0}(A_{n}) → 0. The problem is that A_{n} is not decreasing so we need, somehow, to construct auxiliary decreasing sequence of sets. What does A_{n} → ∅ mean? It means that

G_{n} := ∪_{k≥n}A_{k} ↓ ∅.

If we knew that G_{n} ∈ 𝒞 we would immediately get the result. The problem is we don’t know that, so we can’t use G_{n} as an argument of the function P_{0}. We need to work harder. We only know that finite unions belong to 𝒞. So let us approximate G_{n}. We have

G_{n,r} := ∪_{k=n}^{r}A_{k} ↑ G_{n}, as r → ∞.

We also know that G_{n,r} ∈ 𝒞 for all n, r. So we can talk about P_{0}(G_{n,r}), a numerical sequence which increases with r (and bounded below 1), having a limit, say

γ_{n} = lim_{r→∞} P_{0}(G_{n,r}).

This means that, FOR EACH n, P_{0}(G_{n,r}) differs from γ_{n} by anything we like, when r is large. Let ε_{n} be this “anything we like” thing. We just said that

FOR ALL n, γ_{n} − P_{0}(G_{n,r}) ≤ ε_{n}, for all r larger than or equal to some r(n).

In particular,

FOR ALL n, γ_{n} − P_{0}(G_{n,r(n)}) ≤ ε_{n}.

And if we let

R(n) := max(r(1), r(2), …, r(n), n),

we have

FOR ALL n, γ_{n} − P_{0}(G_{n,R(n)}) ≤ ε_{n}.

Moreover, the sequence R(n) is increasing. We have

A_{n} ⊂ G_{n,R(n)} ⊂ G_{n} ↓ ∅,

so, while we have created a sequence G_{n,R(n)} of sets in 𝒞 which is above A_{n}, and does converge to ∅, we do not know that G_{n,R(n)} decreases, so we cannot apply our assumption (ii). Therefore we need to work harder. We take

H_{n} = ∩_{k=1}^{n}G_{k,R(k)}.

Clearly, H_{n} ∈ 𝒞, and H_{n} decreases, and H_{n} ↓ ∅, therefore, by (ii),

P_{0}(H_{n}) → 0.

But how does H_{n} compare to A_{n}? We lost track of it, so let’s see:

P_{0}(A_{n} \ H_{n}) = P_{0} (∪_{k=1}^{n}(A_{n} \ G_{k,R(k)}))

≤ ∑_{k=1}^{n} P_{0} (A_{n} \ G_{k,R(k)})

≤ ∑_{k=1}^{n} P_{0} (G_{n,R(n)} − G_{k,R(k)})

≤ ∑_{k=1}^{n} (P_{0}(G_{k,R(n)}) − P_{0}(G_{k,R(k)}))

≤ ∑_{k=1}^{n} (ε_{k} − P_{0}(G_{k,R(k)})) ≤ ∑_{k=1}^{n} ε_{k},

where, recall, ε_{k} is ANYTHING WE PLEASE. We would be pleased to show that P_{0}(A_{n} \ H_{n}) → 0, for this would imply P_{0}(A_{n}) → 0. Let us then CHOOSE

ε_{k} = η2^{−k},

where η > 0 is arbitrary. Hence ∑_{k=1}^{n} ε_{k} ≤ η. Hence P_{0}(A_{n} \ H_{n}) ≤ η, and so we can conclude. We proved that (ii) ⇒ (iii). Since, obviously, (iii) ⇒ (ii), the lemma has been proved.

Lemma [COMMON LIMITS] Suppose that P_{0} is countably additive on the field 𝒞.

(i) If A_{n} is a sequence in 𝒞 that has a limit (but which is not necessarily in 𝒞, then P_{0}(A_{n}) has a limit. (ii) If A_{n}, B_{n} are sequences in 𝒞 with the same limit (not necessarily in 𝒞) then P_{0}(A_{n}), P_{0}(B_{n}) have the same limit. Proof Recall that a sequence x_{n} of real numbers converges if and only if, given ε > 0, there is an integer N such that sup_{m>n} |x_{n} − x_{m}| ≤ ε if n > N. The negation of the latter statement implies the existence of ε > 0 such that, for all N there is n > N and m > N with |x_{n} − x_{m}| > ε; or, if we let m = I(n), we have that |x_{n} − x_{I(n)}| ↛ 0, as n → ∞. Therefore, if we can prove that for \underline{all} strictly increasing sequence of integers I(n) we have

x_{n} − x_{I(n)} → 0, as n → ∞,

we can deduce that x_{n} has a limit. Let I(n) be such a sequence. Then

|P_{0}(A_{n}) − P_{0}(A_{I(n)}| ≤ |P_{0}(A_{n}△A_{I(n)}|.

But A_{n} has a limit. Therefore A_{n}△A_{I(n)} → ∅. By the Lemma above, P_{0}(A_{I(n)} → 0, and so (i) Is proved. To prove (ii), we use the same reasoning but with B_{I(n)} instead of A_{I(n)}.

Lemma [FIRST EXTENSION] Suppose that P_{0} is countably additive on the field 𝒞. Let D(𝒞) be the class of subsets of Ω that are limits of sets in 𝒞. Define P_{1} on D(𝒞) by

P_{1}(lim_{n} A_{n}) = lim_{n} P_{0}(A_{n}), A_{n} ∈ 𝒞 .

Then

(i) D(𝒞) is a field. (ii) The definition P_{1} is valid. (iii) P_{1} is countably additive on D(𝒞). Proof (i) Easy. (ii) Follows from Lemma [COMMON LIMITS]. (iii) From Lemma [CA], it suffices to show that if A_{n} is a sequence of elements of D(𝒞) such that A_{n} ↓ ∅ then P_{1}(A_{n}) → 0. Since A_{m} ∈ D(𝒞) we can write

A_{m} = lim_{n} B_{m,n}.

Then we also have

A_{m} = lim_{n}(B_{1,n} ∩ ⋯ ∩ B_{m,n}).

Choose a sequence of integers n_{m} increasing to ∞ such that

lim_{m}[P_{1}(A_{m}) − P_{0}(B_{1,n_{m}} ∩ ⋯ ∩ B_{m,n_{m}})] = 0.

Since

lim_{m}(B_{1,n_{m}} ∩ ⋯ ∩ B_{m,n_{m}}) = ∅,

we have

lim_{m} P_{0}(B_{1,n_{m}} ∩ ⋯ ∩ B_{m,n_{m}}) = 0,

and so

lim_{m} P_{1}(A_{m}) = 0.

Lemma [SECOND EXTENSION] Suppose that P_{0} is countably additive on the field 𝒞. Let D(𝒞) be the class of subsets of Ω that are limits of sets in 𝒞. Let D^{2}(𝒞) be the class of subsets of Ω that are limits of sets in D(𝒞). Define P_{2} on D^{2}(𝒞) by

P_{2}(lim_{n} A_{n}) = lim_{n} P_{1}(A_{n}), A_{n} ∈ D(𝒞).

Then (i) D^{2}(𝒞) is a field. (ii) The definition P_{2} is valid. (iii) P_{2} is countably additive on D^{2}(𝒞). Proof Repetition of the proof of Lemma [FIRST EXTENSION].

Proposition [SANDWICH] Let ℱ be the σ-field generated by 𝒞. Then, for any B ∈ ℱ, there are A, C ∈ D^{2}(𝒞) such that P_{2}(A) = P_{2}(C) and A ⊂ B ⊂ C. Proof Let D^{↓}(𝒞) (respectively, D^{↑}(𝒞)) be the class of sets which are limits of decreasing (respectively, increasing) sequences of sets of 𝒞. Define the class 𝒟 of subsets of Ω for which, given ε > 0, there is G ∈ D^{↓}(𝒞) and H ∈ D^{↑}(𝒞) such that

G ⊂ A ⊂ H, P_{1}(H) − P_{1}(G) < ε.

We have 𝒞 ⊂ 𝒟. Also, Ω ∈ 𝒟. So if we show that 𝒟 is both π and λ, then it is σ and so σ(𝒞) ⊂ 𝒟 and we are done. That it is closed under intersections and proper differences is easy. That it is closed under increasing limits is left as an exercise.

Probability

4: Joint distributions and independence

Takis Konstantopoulos^{a}

             www.smstc.ac.uk

\underline{Contents}

4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–1 4.2 Joint distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–1

4.2.1 Knowledge of F_{X,Y} implies knowledge of P_{X,Y} . . . . . . . . . . . . . . 4–2 4.3 Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–3 4.4 Joint density . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–4 4.5 Joint moment generating function . . . . . . . . . . . . . . . . . . . . 4–5 4.6 Correlations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–5 4.7 Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–7

4.7.1 Naïve conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–7 4.7.2 Geometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–7 4.7.3 Conditional probability . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–9 4.7.4 Conditional law . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–10

4.8 Gaussian variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–12 4.8.1 The Gaussian law . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–12 4.8.2 The multidimensional Gaussian random variable . . . . . . . . . . . . . 4–13 4.8.3 Conditional Gaussian law . . . . . . . . . . . . . . . . . . . . . . . . . . 4–16

4.1 Introduction

Most of this chapter is concerned with a random variable (X, Y) with values in ℝ^{2}, but the concepts/results are easily generalisable to random variables with values in ℝ^{d}.

We first explain why the joint distribution function defines the law of (X, Y). We talk about independence and carefully explain why independence between classes of sets closed under intersection implies independence between their σ-fields.

To define conditional density we define projection and prove everything, except the existence of a regular conditional distribution.

Finally, we talk about a Gaussian variable in ℝ^{d}.

4.2 Joint distributions

Consider a random variable (X, Y) : (Ω, ℱ) → (ℝ^{2}, ℬ(ℝ^{2})). This random variable is called TWO RANDOM VARIABLES.

4–1 If P is a probability of (Ω, ℱ), the joint distribution of (X, Y) is another name for the law P_{X,Y} of the random variable (X, Y).

The joint distribution function of (X, Y) is the function

F_{X,Y} (x, y) := P(X ≤ x, Y ≤ y) = P_{X,Y} ((−∞, x] × (−∞, y]), (x, y) ∈ ℝ^{2}.  (4.1)

The law P_{X} of X is referred to as the first marginal of the law P_{X,Y}. The distribution function F_{X} of X is referred to as the first marginal distribution function of the joint distribution function F_{X,Y} and, of course,

F_{X}(x) = lim_{y→∞} F_{X,Y} (x, y).

Note that we chose ti use ≤ instead of < in (4.1) for no good reason other than a mere arbitrary convention.

4.2.1 Knowledge of F_{X,Y} implies knowledge of P_{X,Y}

We would like to explain why, knowledge of the function F_{X,Y} implies knowledge of P_{X,Y} (B) for all B ∈ ℬ(ℝ^{2}). Consider a rectangle (with sides parallel to the axes–please think geometrically)

(a_{1}, b_{1}] × (a_{2}, b_{2}] := {(x, y) ∈ ℝ^{2} : a_{1} < x ≤ b_{1}, a_{2} < y ≤ b_{2}}.  (4.2)

We allow a_{1}, a_{2} to take any value, including −∞. Since (−∞, b_{1}]×(−∞, b_{2}] is the disjoint union of four rectangles, using additivity, we obtain

P_{X,Y} ((a_{1}, b_{1}] × (a_{2}, b_{2}]) = F_{X,Y} (b_{1}, b_{2}) − F_{X,Y} (b_{1}, a_{2}) − F_{X,Y} (a_{1}, b_{2}) + F_{X,Y} (b_{1}, b_{2}).

Rectangles of the form (4.2) have the following nice properties: First, intersection of two of them is a rectangle of the same form. Thus, if ℛ denotes the collection of these rectangles we have that ℛ is closed under finite intersection, i.e. it is a π-system. Second, the complement of a rectangle from ℛ is a finite union of disjoint rectangles from ℛ.

EXERCISE 31. Write, explicitly, the complement of (a_{1}, b_{1}] × (a_{2}, b_{2}] as the disjoint union of elements of ℛ. Notice we can do that in at least two different ways. Do so.

Now consider the class

𝒞 := {finite disjoint unions of elements of ℛ }.

It is easy to visualise, geometrically, what kind of elements 𝒞 contains. Then

EXERCISE 32. Show that 𝒞 is a field, i.e. if A ∈ 𝒞 then A^{c} ∈ 𝒞 and if A, B ∈ 𝒞 then A ∪ B ∈ 𝒞.

Note that the notation used here mimics that of Section 3.7. Indeed, the classes ℛ, 𝒞 here behave in the same way as the classes with the same names there.

Hence, if A ∈ 𝒞, we can write A = ∪_{i=1}^{m}B_{i}, where B_{i} are disjoint rectangles from ℛ, and, since for each such rectangle B_{i} we can use F_{X,Y} to compute P_{X,Y} (B), we have that

P_{X,Y} (A) = ∑_{i=1}^{m} P_{X,Y} (B), A ∈ 𝒞 ,

can be computed by using F_{X,Y} only.

But there are many sets in ℬ(ℝ^{2}) that do not belong to 𝒞, so we wish to continue our endeavour. If it were true that every set in ℬ(ℝ^{2}) was a limit of sets of 𝒞 then we would be finished, by the continuity property of a probability. However, there are many elements in ℬ(ℝ^{2}) that are not limits of sets in 𝒞.

Example 4.1. The set of all points (x, y) where x, y are rationals cannot be obtained as a limit of elements of 𝒞^{2}. To illuminate this point, consider “straightforward” procedure that places a little rectangle around each such point and then let the little rectangle shrink. Specifically, let ℚ = {q_{1}, q_{2}, …} be an enumeration of the rationals. To each (q_{m}, q_{n}) associate the rectangle

I_{m,n}(ε) := (q_{m} − ε2^{−m}, q_{m} + ε2^{−m}] × (q_{n} − ε2^{−n}, q_{n} + ε2^{−n}],

and let I(ε) := ∪_{m,n}I_{m,n}(ε). Show that ∩_{ε>0,ε∈ℚ}I(ε) is not equal to ℚ × ℚ. Hint: The set I(ε) is uncountable.

We will denote by P_{0} the function P_{X,Y} restricted to 𝒞. Clearly, P_{0} is uniquely specified by F_{X,Y}. If we show that P_{0} can be uniquely extended to a probability on (ℛ^{2}, ℬ(ℝ^{2})) then our claim that F_{X,Y} completely specifies P_{X,Y} will be proved. The steps in showing this are precisely the same as those of Section 3.7.

1. We first show that P_{0} is countably additive on 𝒞.

2. We then extend P_{0} to P_{1} on D(𝒞).

3. We then extend P_{1} to P_{2} on D^{2}(𝒞).

4. Finally, we show, precisely as before, that, for every B ∈ ℬ(ℛ^{2}) there are sets A, C ∈ D^{2}(𝒞) such that A ⊂ B ⊂ C and P_{2}(A) = P_{2}(C).

4.3 Independence

Recall (see 2.2 ) that X, Y are independent random variables on (Ω, ℱ P) if σ(X), σ(Y) are independent σ-fields. We wish to show that

Proposition 4.1. X, Y are independent on (Ω, ℱ , P) if and only if F_{X,Y} (x, y) := P(X ≤ x, Y ≤ Y), F_{X}(x) := P(X ≤ x), F_{Y} (y) := P(Y ≤ y) are related by

F_{X,Y} (x, y) = F_{X}(x)F_{Y} (y).

This is a consequence of the following:

Lemma 4.1. Let (Ω, ℱ , P) be a probability space. Let ℛ_{1}, ℛ_{2} be two π-systems. Then σ(ℛ_{1}), σ(ℛ_{2}) are independent if and only if

P(B_{1}B_{2}) = P(B_{1})P(B_{2}), B_{1} ∈ ℛ_{1}, B_{2} ∈ ℛ_{2}.

Proof The proof mimics the uniqueness part of the proof of Theorem 3.1. Fix B_{1} ∈ ℛ_{1} and consider two probabilities on σ(ℛ_{2}):

P(B_{1}A_{2}), P(B_{1})P(A_{2}), A_{2} ∈ σ(ℛ_{2}).

Our assumption says that these two probabilities agree on ℛ_{2}. Since ℛ_{2} is a π-system, the two probabilities agree on σ(ℛ_{2}): For each B_{1} ∈ ℛ_{1},

P(B_{1}A_{2}) = P(B_{1})P(A_{2}), A_{2} ∈ σ(ℛ_{2}).  (4.3)

Now, for fixed A_{2} ∈ σ(ℛ_{2}) consider two probabilities on σ(ℛ_{1}):

P(A_{1}A_{2}), P(A_{1})P(A_{2}), A_{1} ∈ σ(ℛ_{1}).

(4.3) says that these two probabilities agree on ℛ_{1}. Since ℛ_{1} is a π-system, the two probabilities agree on σ(ℛ_{1}): For each A_{2} ∈ σ(ℛ_{2}),

P(A_{1}A_{2}) = P(A_{1})P(A_{2}), A_{1} ∈ σ(ℛ_{1}),

which is what we need. □

Proof of Proposition 4.1: Let ℛ_{1} = {X^{−1}(a, b] : a < b}, ℛ_{2} = {Y^{−1}(a, b] : a < b}. These are both π-systems. It is easy to see that the assumption F_{X,Y} (x, y) = F_{X}(x)F_{Y} (y) implies that P(B_{1}B_{2}) = P(B_{1})P(B_{2}), B_{1} ∈ ℛ_{1}, B_{2} ∈ ℛ_{2}. Hence P(A_{1}A_{2}) = P(A_{1})P(A_{2}), A_{1} ∈ σ(ℛ_{1}) = σ(X), A_{2} ∈ σ(ℛ_{2}) = σ(Y).

4.4 Joint density

Consider the coin flip space (Ω = {0, 1}^{ℕ}, ℱ , P). Define the random variables

U_{1} := ∑_{n=1}^{∞} \frac{ω_{2n}}{2^{n}} , U_{2} := ∑_{n=1}^{∞} \frac{ω_{2n−1}}{2^{n}} .

By applying Lemma 4.1 we can see that U_{1}, U_{2} are independent.

Let h : (ℝ^{2}, ℬ(ℝ^{2})) → (ℝ, ℬ) be measurable and suppose h ≥ 0. Define the Lebesgue integral

∫_{ℝ^{2}} h(x, y)dxdy := ∑_{n_{1},n_{2}∈ℤ} Eh(U_{1} + n_{1}, U_{2} + n_{2}),

and, more generally, for any B ∈ ℬ(ℝ^{2}),

∫_{B} h(x, y)dxdy := ∫_{ℝ^{2}} h(x, y)1_{B}(x, y)dxdy.

If h has no restriction on sign, define ∫_{B} h(x, y)dxdy := ∫_{B} h^{+}(x, y)dxdy − ∫_{B} h^{−}(x, y)dxdy, provided not both terms are infinite. The following theorem connects what one learns in basic Calculus of Many Variables with what we just defined.

Theorem 4.1. (i) If h is Riemann integrable on a rectangle R = [a, b] × [c, d] then its Lebesgue integral on R coincides with its Riemann integral on R. (ii) If h is bounded and measurable then it is Riemann integrable on R of the set of discontinuities D of h satisfies P((U_{1}, U_{2}) ∈ D) = 0.

More generally, given two distribution functions F_{1}, F_{2} on ℝ, we can define the product measure F_{1} × F_{2} on (ℝ^{2}, ℬ(ℝ^{2})) by defining it first on rectangles,

(F_{1} × F_{2})(B_{1} × B_{2}) := P(F_{1}^{−1}(U_{1}) ∈ B_{1}, F_{2}^{−1}(U_{2}) ∈ B_{2}),

and then extending it to ℬ(ℝ^{2}) using the procedure explained earlier. We can also define the Lebesgue-Stieltjes integral

∫_{ℝ^{2}} h d(F_{1} × F_{2}) := Eh(F_{1}^{−1}(U_{1}), F_{2}^{−1}(U_{2})).

Theorem 4.2 (Fubini). If h ≥ 0 or if ∫_{ℝ^{2}} |h| d(F_{1} × F_{2}) < ∞, then

∫_{ℝ^{2}} h d(F_{1} × F_{2}) = ∫_{ℝ} (∫_{ℝ} h(x, y)F_{2}(dy)) F_{1}(dx) = ∫_{ℝ} (∫_{ℝ} h(x, y)F_{1}(dx)) F_{2}(dy).

Proof [sketch]: The statement is obvious when h(x, y) = 1_{B_{1}}(x)1_{B_{2}}(y). Hence it is obvious for linear combinations of such functions. The general statement follows from an approximation procedure relying on an argument based on the Sierpiński-Dynkin lemma.

Occasionally, it so happens that there exists a function f such that F_{X,Y} can be written as a Lebesgue integral:

F_{X,Y} (x, y) = ∫_{(−∞,x]×(−∞,y]} f(s, t)dsdt.

In such a case, we say that (X, Y) is absolutely continuous (jointly absolutely continuous, I suppose, if you want to be pedantic) and that f_{X,Y} is a density. Using Fubini’s theorem, we see that

f_{X}(x) = ∫_{(−∞,x]} f(s)ds

is a density for X (i.e. X is absolutely continuous, and so is Y).

Another consequence of Fubini’s theorem is:

Lemma 4.2. If X, Y are independent then

E(XY) = (EX)(EY),

whenever the expectations are defined.

And another, useful consequence of Fubini’s theorem is:

Lemma 4.3. If X is a positive random variable then

EX = ∫_{0}^{∞} P(X > x) dx.

A standard criterion for independence between X, Y, for absolutely continuous (X, Y), is:

Lemma 4.4. Suppose that (X, Y) is absolutely continuous. Let f_{X,Y} be a density of (X, Y). Let f_{X}, f_{Y} be densities of X, Y, respectively. Then X, Y are independent if and only if

f(x, y) = f_{X}(x)f_{Y} (y),

for all (x, y) except, possibly, on a set of measure zero.

4.5 Joint moment generating function

When (X, Y) is a random variable in ℝ^{2} we can define its moment generating function by

M_{X,Y} (η, θ) := Ee^{ηX+θY} , η, θ ∈ ℝ.

whenever it exists. Let also M_{X}, M_{Y} be the moment generating functions of X, Y. One can prove that:

Lemma 4.5. Suppose that M_{X,Y} exists in a (one sided) neighbourhood of zero. If M_{X,Y} (η, θ) = M_{X}(η)M_{Y} (θ) then X, Y are independent.

4.6 Correlations

We now consider random variables X on some probability space (Ω, ℱ , P) with EX^{2} < ∞. The aggregate of all these random variables will be denoted by

L^{2}(Ω, ℱ , P).

If EX^{2} < ∞, EY^{2} < ∞, then E(X + Y)^{2} < ∞ (indeed, (x + y)^{2} ≤ 2x^{2} + 2y^{2}). This means that if X, Y ∈ L^{2}(Ω, ℱ , P) then, for any a, b ∈ ℝ, aX + bY ∈ L^{2}(Ω, ℱ , P). Hence L^{2}(Ω, ℱ , P) is a linear space (a linear subspace of ℝ^{Ω}). Recall that the covariance between X and Y is defined by

cov(X, Y) = E(X − EX)(Y − EY).

Since, for X_{1}, X_{2}, Y ∈ L^{2}(Ω, ℱ , P), a_{1}, a_{2} ∈ ℝ,

cov(a_{1}X_{1} + a_{2}X_{2}, Y) = a_{1} cov(X_{1}, Y) + a_{2} cov(X_{2}, Y),

the covariance is linear in each of its arguments when the other is kept fixed and it can thus be used to define an inner product:

⟨X, Y ⟩ := cov(X, Y).

We also define the semi-norm

||X|| := \sqrt{cov(X, X)} = \sqrt{var(X, X)},

where the word ‘semi-norm’ means that it has the following properties:

1. ||X|| ≥ 0

2. ||a_{1}X_{1} + a_{2}X_{2}|| = |a_{1}| ||X_{1}|| + |a_{2}| ||X_{2}||.

3. ||X + Y || ≤ ||X|| + ||Y ||.

Another name for ||X|| is ‘standard deviation’. We also note that

If ||X|| = 0 then P(X = 0) = 1.

We \underline{cannot} deduce, from ||X|| = 0 alone that X(ω) = 0 for all ω ∈ Ω, but only that X(ω) = 0 for all ω except those in a set of probability zero. If we could deduce that X(ω) = 0 for all ω ∈ Ω, we would say that || · || is a norm. To get around this problem, we merely identify all random variables in L^{2}(Ω, ℱ , P) which differ on a set of measure zero: That is, we let [X] be the set of all Y such that P(X ≠ Y) = 0, and redefine L^{2}(Ω, ℱ , P) to be the collection of all such [X]. It is not hard to see that this is still a linear space and if we let ||[X]|| := ||X|| (which is well defined), then this is a norm.

Being a normed space with an inner product, L^{2}(Ω, ℱ , P) has a structure much like the geometric structure of the usual Euclidean space, for instance, Pythagoras’ theorem holds:

||X + Y ||^{2} = ||X||^{2} + ||Y ||^{2} if ⟨X, Y ⟩ = 0.

The notion of convergence in L^{2}(Ω, ℱ , P) is as follows: We say that X_{n} → X if ||X_{n} − X|| → 0, as n → ∞. With respect to this notion of convergence, the space L^{2}(Ω, ℱ , P) is complete:

Lemma 4.6. If X_{n} is a sequence in L^{2}(Ω, ℱ , P) such that

sup_{m,n≥N} ||X_{n} − X_{m}|| → 0 as N → ∞,

then there is X ∈ L^{2}(Ω, ℱ , P) such that

||X_{m} − X|| → 0 as m → ∞.

Proof The assumption means that, given any ε > 0, we can find and index N(ε) such that ||X_{n} − X_{m}|| ≤ ε for all n, m ≥ N(ε). Since we can do this for any ε, we can do it for ε = 2^{−1}, 2^{−2}, 2^{−3}, . . .. Thus, if ε = 2^{−ℓ}, there is an index N_{ℓ} such that

||X_{n} − X_{m}|| ≤ 2^{−ℓ} for all n, m ≥ N_{ℓ}.  (4.4)

Define M_{ℓ} = max(N_{1}, N_{2}, …, N_{ℓ}). Then ||X_{M_{ℓ+1}} − X_{M_{ℓ}}|| ≤ 2^{−ℓ}. But then

E|X_{M_{ℓ+1}} − X_{M_{ℓ}}| ≤ ||X_{M_{ℓ+1}} − X_{M_{ℓ}}|| ≤ 2^{−ℓ}.

Hence

∑_{ℓ=1}^{∞} E|X_{M_{ℓ+1}} − X_{M_{ℓ}}| ≤ 1,

and, by Fubini’s theorem,

E ∑_{ℓ=1}^{∞} |X_{M_{ℓ+1}} − X_{M_{ℓ}}| < ∞.

Therefore, the random variable sum_{ℓ=1}^{∞}|X_{M_{ℓ+1}} − X_{M_{ℓ}}| is finite with probability 1. Put it otherwise, there is a set A ∈ ℱ, with P(A) = 0, such that

∑_{ℓ=1}^{∞} |X_{M_{ℓ+1}}(ω) − X_{M_{ℓ}}(ω)| < ∞, ω ∈ A^{c}.

This means that

lim_{n→∞} ∑_{ℓ=1}^{n−1}(X_{M_{ℓ+1}}(ω) − X_{M_{ℓ}}(ω)) exists for all ω ∈ A^{c}.

But the sum is just X_{M_{n}}(ω) − X_{M_{1}}(ω). We have thus proved that for all ω ∈ A^{c} there is X(ω) such that X_{M_{n}}(ω) → X(ω) as n → ∞. Since this is a limit, the function X is a random variable. Fix ℓ and use (4.4) again with m ≥ M_{ℓ}, n = M_{r}, for r ≥ ℓ:

E|X_{m} − X_{M_{r}}|^{2} ≤ 4^{−ℓ}, m ≥ M_{ℓ}, r ≥ ℓ.

By Fatou’s lemma, for all m ≥ M_{ℓ},

E|X_{m} − X|^{2} = E \underline{lim}_{r→∞} |X_{m} − X_{M_{r}}|^{2} ≤ \underline{lim}_{r→∞} E|X_{m} − X_{M_{r}}|^{2} ≤ 2^{−ℓ}.

Since ||X_{m} − X|| ≥ ||X|| − ||X_{m}||, we have ||X|| ≤ ||X_{m}|| + 2^{ℓ} < ∞, hence X ∈ L^{2}(Ω, ℱ , P). Also, lim_{m→∞} ||X_{m} − X|| = 0, as required. □

4.7 Conditioning

4.7.1 Naïve conditioning

Naïve conditioning was defined in 2.1 . Namely, if (Ω, ℱ , P) is a probability space and B ∈ ℱ, P(B) ≠ 0, we let

P(A|B) = P(AB)/P(B), A ∈ ℱ ,

be a new probability, called P conditional on B. We easily see that (Ω, ℱ , P(·|B)) is a new probability space.

The problem is that, in many cases, we want to condition with respect to an event B that has probability zero. This can be done. Loosely speaking, what saves us is the fact that if P(B) = 0 then P(AB) = 0 as well.

We can use many methods for defining conditioning in a more general sense. We adopt a geometric approach. In Chapter 8 you will see another approach.

4.7.2 Geometry

Consider the Euclidean space ℝ^{d}. Let 𝔾 be a linear subspace of it. Let ||x − y|| stand for the Euclidean distance between x, y ∈ ℝ^{d}, i.e.

||x − y|| := \sqrt{(x_{1} − y_{1})^{2} + ⋯ + (x_{d} − y_{d})^{2}},

and

⟨x, y⟩ := x_{1}y_{1} + ⋯ + x_{d}y_{d}.

Thus ⟨x, y⟩/||x|| ||y|| is the cosine of the angle between x and y. We wish to project an x ∈ ℝ^{d} onto 𝔾:

Lemma 4.7. Given any x ∈ ℝ^{d} there is a unique x̂ ∈ 𝔾 such that

||x − x̂|| = min_{y∈𝔾} ||x − y||.

If we think geometrically, the answer is obvious: this x̂ is such that x − x̂ is orthogonal to 𝔾; in other words, the angle between x − x̂ and any y ∈ 𝔾 is a right angle, i.e. its cosine is zero:

⟨x − x̂, y⟩ = 0, y ∈ 𝔾.

Equivalently,

⟨x̂, y⟩ = ⟨x, y⟩, y ∈ 𝔾.  (4.5)

Since 𝔾 is a linear space, (4.5) defines x̂ uniquely. To see this directly, assume that there are two elements x̂, x′ ∈ 𝔾 satisfying (4.5). Then ⟨x̂, y⟩ = ⟨x′, y⟩ for all y ∈ 𝔾. Thus, ⟨x̂ − x′, y⟩ = 0, for all y ∈ 𝔾. In particular, for y = x̂ − x′, we have ⟨x̂ − x′, x̂ − x′⟩ = 0, which immediately gives x̂ = x′. Proof of Lemma 4.7: Define (uniquely) x̂ ∈ 𝔾 by (4.5). Obviously,

||x̂ − y||^{2} = ⟨x̂ − y, x̂ − y⟩ ≥ 0, y ∈ 𝔾.

We have

0 ≤ ⟨x̂ − y, x̂ − y⟩ = ||x̂||^{2} + ||y||^{2} − 2⟨x̂, y⟩
= ||x̂||^{2} + ||y||^{2} − ⟨x, y⟩ − ⟨x̂, y⟩, y ∈ 𝔾,

where we used (4.5). We now use two things. First,

||x − y||^{2} = ||x||^{2} + ||y||^{2} + 2⟨x, y⟩,

and second,

||x||^{2} = ||(x − x̂) + x̂||^{2} = ||x − x̂||^{2} + ||x̂||^{2} + 2⟨x − x̂, x⟩ = ||x − x̂||^{2} + ||x̂||^{2},

because ⟨x − x̂, x⟩ = 0, by (4.5). Combining the last two displays our inequality becomes

0 ≤ ||x − y||^{2} − ||x − x̂||^{2}, y ∈ 𝔾,

which is what we want.

Notation: We will write

x̂ = proj(x|𝔾),

to remind ourselves that x̂ is obtained by projecting x onto 𝒢.

EXERCISE 33. Let ℍ be a linear subspace of 𝔾. Then

proj ( proj(x | 𝔾) | ℍ) = proj(x | ℍ).

(This should be geometrically ‘obvious’: it is an exercise in right angles.)

4.7.3 Conditional probability

Suppose now that 𝒢 ⊂ ℱ is a σ-field. Then L^{2}(Ω, 𝒢 , P) is a linear subspace of L^{2}(Ω, ℱ , P). By analogy to the geometric picture above, given X ∈ L^{2}(Ω, ℱ , P), we wish to define X̂ ∈ L^{2}(Ω, 𝒢 , P) such that

||X − X̂|| = min_{Y ∈L^{2}(Ω,𝒢 ,P)} ||X − Y ||.

Owing to the structure of L^{2}(Ω, 𝒢 , P), this can be done:

Theorem 4.3. Let 𝒢 ⊂ ℱ be a σ-field. Given X ∈ L^{2}(Ω, ℱ , P), there exists X̂ ∈ L^{2}(Ω, 𝒢 , P) such that one of the following two equivalent statements holds:

(i)

||X − X̂|| = inf_{Y ∈L^{2}(Ω,𝒢 ,P)} ||X − Y ||.  (4.6)

(ii)

EXY = EX̂Y, for all Y ∈ L^{2}(Ω, 𝒢 , P).  (4.7)

Furthermore, X̂ is almost surely unique, in the sense that if X̂′ ∈ L^{2}(Ω, 𝒢 , P) satisfies (i) or (ii) then P(X̂ = X̂′) = 1.

Proof [sketch]: We only need to prove existence of such an X̂. The rest of the proof is as in the previous section. Existence is guaranteed by Lemma 4.4. Indeed, by the definition of inf, we can choose Y_{n} ∈ L^{2}(Ω, 𝒢 , P) such that ||X − Y_{n}|| converges to the infimum in (4.6). It is easy to see that sup_{m,n≥N} ||Y_{n} − Y_{m}|| → 0 as N → ∞, therefore, by Lemma 4.4, we have an X̂ ∈ L^{2}(Ω, 𝒢 , P) such that ||Y_{n} − X̂|| → 0 as n → ∞. This X̂ does the job.

We use the following notation:

X̂ = E_{P}(X|𝒢).

The notation reminds us that X̂ depends both on 𝒢 and on P. If P is implicitly understood then we just write X̂ = E(X|𝒢).

The next exercise justifies using the symbol E for this projection. Indeed, if the σ-field 𝒢 is the smallest possible, then the projection is a constant, the expectation of X:

EXERCISE 34. Show that E(X|{∅, Ω}) = EX. (Hint: E(X − t)^{2} is minimised by t = EX.)

What happens if X is 𝒢 measurable?

EXERCISE 35. If X ∈ L^{2}(Ω, 𝒢 , P) then E(X|𝒢) = X.

We can also show the double projection property:

EXERCISE 36. If ℋ ⊂ 𝒢 ⊂ ℱ are all σ-fields then

E(E(X | 𝒢) | ℋ ) = E(X | ℋ ).

Suppose now that A ∈ ℱ. Then we define

P(A | 𝒢) := E(1_{A} | 𝒢),

and call this conditional probability of the event A given the σ-field 𝒢. By definition, for each A ∈ ℱ, the conditional probability P(A | 𝒢) is a random variable, specifically it is 𝒢 -measurable. Also, by linearity, if A_{1}A_{2} = ∅,

P(A_{1} ∪ A_{2}|𝒢) = P(A_{1} | 𝒢) + P(A_{2} | 𝒢),

except on a set of probability zero. The question is: Is ℱ ∋ A ↦ P(A | 𝒢) a probability for almost every ω ∈ Ω? The answer is not obvious because P(A | 𝒢) is defined uniquely up to an event N_{A} of probability zero. Since there are uncountably many A’s in ℱ, it is not clear we can simultaneously define all the random variables P(A | 𝒢) up to an event of probability zero. In special cases this can be done.

4.7.4 Conditional law

Here is one such special case: Let X, Y be random variables on (Ω, ℱ , P) with finite second moments. We define

P(X ∈ B | Y) := P(X^{−1}(B) | σ(Y)), B ∈ ℬ.

While we can certainly do this rigorously for each fixed B ∈ ℬ, it is not immediate that for each ω ∈ Ω, this is a probability in B. It can be shown that it actually is, in the sense that we can choose P(X ∈ B | Y) so that it is a probability as a function of B and simultaneously a measurable function of Y. We call this object conditional law of X given Y.

The conditional distribution function of X given Y is defined as

P(X ≤ x | Y).

To be honest with the definitions, we must make sure they reduce to the usual naïve ones from elementary probability. We consider two cases.

Case 1: (X, Y) is absolutely continuous

Lemma 4.8. Suppose that (X, Y) is absolutely continuous with joint density f(x, y). Let f_{2}(y) = ∫_{ℝ} f(x, y)dx be the density of Y. Define the conditional density of X given Y by

f_{X|Y} (x|y) := \begin{cases}\frac{f(x, y)}{f_{2}(y)}, & if f_{2}(y) ≠ 0, \\ 0, & otherwise. \end{cases}

For B ∈ ℬ, let

g_{B}(y) := ∫_{B} f_{X|Y} (x|y)dx.

Then

P (X ∈ B|Y) = g_{B}(Y),

for all ω except on a set of probability zero.

Proof We need to show that (see (4.7))

E1(X ∈ B)W = Eg_{B}(Y)W,

for all W ∈ L^{2}(Ω, σ(Y), P). It is enough to show this for al, W of the form W = 1(Y ∈ C), C ∈ ℬ:

E1(X ∈ B)1(Y ∈ C) = Eg_{B}(Y)1(Y ∈ C).

But the left hand side equals

∫_{ℝ^{2}} 1_{B}(x)1_{C}(y)f_{X,Y} dxdy.

The right hand side equals

∫_{R} g_{B}(y)1_{C}(y)dy.

These two quantities are the same, by Fubini’s theorem. □

Case 2: (X, Y) is discrete

Suppose that (X, Y) takes values in a discrete set S_{1} × S_{2}. As usual, define its probability mass function and marginal mass functions by

p(x, y) := P((X, Y) = (x, y)), p_{1}(x) := P(X = x), p_{2}(y) := P(Y = y), (x, y) ∈ S_{1} × S_{2}.

We then have a simple formula for the conditional law of X given Y, precisely the one we expect:

Lemma 4.9. Let (X, Y) be a random variable on (Ω, ℱ , P) with values in the discrete set (S_{1} × S_{2}, 2^{S_{1}×S_{2}}). Define the conditional probability mass function of X given Y by

p_{X|Y} (x|y) := \begin{cases}\frac{p(x, y)}{p_{2}(y)}, & if p_{2}(y) ≠ 0, \\ 0, & otherwise. \end{cases}

Let P(X = x|Y) be defined as the projection of 1(X = x) onto L^{2}(Ω, σ(Y), P). Then^{a}

P(X = x|Y) = p_{X|Y} (x|Y).

Proof We need to verify that p_{X|Y} (x|Y) satisfies (4.7), namely,

E1(X = x)W = Ep_{X|Y} (x|Y)W, W ∈ L^{2}(Ω, σ(Y), P).

We work with the case where S_{2} is a finite set, the more general case being similar. We can then write Y as a finite sum of the form

Y = ∑_{y} y1(Y = y)

where the A_{k} are mutually disjoint with P(A_{k}) > 0. Any W which is σ(Y)-measurable is a function of Y, and so we can take

W = ∑_{y} g(y)1(Y = y).

We have

E[p_{X|Y} (x|Y)W] = ∑_{y} g(y)E[p_{X|Y} (x|Y)1(Y = y)]

= ∑_{y} g(y)E[p_{X|Y} (x|y)1(Y = y)] = ∑_{y} g(y)p_{X|Y} (x|y)P(Y = y).

If P(Y = y) = 0 then p_{X|Y} (x|y) = 0. Otherwise, p_{X|Y} (x|y) = p(x, y)/p_{2}(y), and so

E[p_{X|Y} (x|Y)W] = ∑_{y} g(y)p(x, y) = EXW,

as required. □

aThe notation p_{X|Y} is terrible. We only use it out of some respect to the undergraduate probability courses.

The reason that the notation is terrible is that in the subsript ‘X|Y ’ in p_{X|Y} (x|Y) plays a merely cosmetic rôle, as opposed to the essential rôle played by the last variable Y inside the parenthesis.

4.8 Gaussian variables

4.8.1 The Gaussian law

The motivation of the Gaussian probability comes from the central limit theorem (which, long time ago, was known as the “law of errors”). This was stated, without proof, in 2.3.2 . We work heuristically in order to motivate the definitions. Let S_{n} = ξ_{1} + ⋯ + ξ_{n} be the sum of n independent indicator (also known as Bernoulli) random variables ξ_{i} with P(ξ_{i} = 1) = p for all i. Let Ŝ_{n} = S_{n} − ES_{n}. Then, as n → ∞, the distribution function of Ŝ_{n}/\sqrt{n} converges to an absolutely continuous distribution function with a famous formula. Let X be a random variable with such a distribution function. Note that

\frac{Ŝ_{2n}}{\sqrt{n}} = \frac{1}{\sqrt{2}} \frac{Ŝ_{n}}{\sqrt{n}} + \frac{1}{\sqrt{2}} \frac{Ŝ′_{n}}{\sqrt{n}},

where Ŝ′_{n} = ξ_{n+1} + ⋯ + ξ_{2n} is a random variable with the same law as Ŝ_{n}. So the distribution of \frac{Ŝ′_{n}}{\sqrt{n}} will also converge to the distribution of X. Moreover, Ŝ′_{n}, Ŝ_{n} are independent. Therefore, if X_{1}, X_{2} are independent random variables with the same law as X, then we expect that

X = \frac{X_{1} + X_{2}}{\sqrt{2}} .

Even if we do not know what this famous distribution is, it should be such that this “addition law” holds. From this, we can discover its density. One way to do that is by imposing the extra assumption (which is not necessary) that the generating function of X exists for all θ: M(θ) = Ee^{θX}. Then

M(θ) = M(θ/\sqrt{2})^{2}.

Letting θ = \sqrt{η} and taking logarithms, we have

log M(\sqrt{η}) = 2 log M(\sqrt{η/2}).

So, if we temporarily let m(η) = log M(\sqrt{(}η)) we have

m(η) = 2m(η/2).

So m(0) = 0 and with some work, we can actually find that the only continuous function satisfying the latter is linear: m(η) = cη. (This should be geometrically obvious.) Hence

M(θ) = e^{cθ^{2}}.

From Lemma 3.32 we know that the moments of X are given by the derivatives of M at 0. We find

M′(0) = 0, M′′(0) = 2c.

So EX = 0 (as it should), while EX^{2} = 2c. So c > 0. Since EX = 0, the second moment is the variance and we customarily denote it by σ^{2}. We arrive at

M(θ) = e^{\frac{1}{2} σ^{2}θ^{2}}.

We know that there is only one probability distribution with a given moment generating function. Instead of trying to figure out which one it is, let us reveal the result and then just verify that it is correct. We claim that the probability distribution corresponding to the last M is absolutely continuous with density

f(x) = Ce^{−x^{2}/2σ^{2}}.

Here, C is such that ∫_{ℝ} f(x)dx = 1. This is the famous normal or Gaussian density with mean 0 and variance σ^{2}. It is called standard if σ^{2} = 1.

EXERCISE 37. Show that ∫_{−∞}^{∞} e^{θx}f(x)dx = e^{\frac{1}{2} σ^{2}θ^{2}}. (Hint: Complete the square and use the definition of C.)

Now, to find C is a very important thing. It is based on the following:

Lemma 4.10.

∫_{ℝ} e^{−x^{2}}dx = \sqrt{π}.

(Liouville said that a Mathematician is someone for whom this integral is obvious.)

EXERCISE 38. Use Fubini’s theorem to write (∫_{ℝ} e^{−x^{2}}dx)^{2} = ∫_{ℝ^{2}} e^{−x^{2}−y^{2}}dxdy and do the latter integral using polar coördinates.

Using this we find that

C = \frac{1}{\sqrt{2πσ^{2}}} .

Therefore, the standard normal density is

φ(x) = \frac{1}{\sqrt{2π}} e^{−x^{2}/2}.

We write N (0, 1) for the law of a random variable X with standard normal density. We write N (μ, σ^{2}) for the law of σX + μ.

EXERCISE 39. Show that a density for N (μ, σ^{2}) is

\frac{1}{\sqrt{2πσ^{2}}} exp \{−\frac{(x − μ)^{2}}{2σ^{2}} \} .

EXERCISE 40. Show that if X_{i}, i = 1, …, d are independent and X_{i} having law N (μ_{i}, σ_{i}^{2}) then ∑_{i=1}^{d} X_{i} has law N (∑_{i} μ_{i}, ∑_{i} σ_{i}^{2}). Therefore linear combinations of independent normal variables are normal.

4.8.2 The multidimensional Gaussian random variable

We now pass on to defining a Gaussian (or normal) variable in ℝ^{d} (i.e. a random vector).

We say that (X_{1}, …, X_{d}) is Gaussian in ℝ^{d} if, for all a_{1}, …, a_{d} ∈ ℝ, the random
variable a_{1}X_{1} + ⋯ + a_{d}X_{d} is normal.

The next lemma shows what the moment generating function of a normal vector is:

Lemma 4.11. If (X_{1}, …, X_{d}) is Gaussian vector with

μ_{j} = EX_{j}, r_{jk} = cov(X_{j}, X_{k}),

then

E exp ∑_{j=1}^{d} θ_{j}X_{j} = exp \{∑_{j=1}^{d} μ_{j}θ_{j} + \frac{1}{2} ∑_{j=1}^{d} ∑_{k=1}^{d} r_{jk}θ_{j}θ_{k}\} .

Proof By definition ∑_{j=1}^{d} θ_{j}X_{j} should be normal, i.e. have a law N (μ, σ^{2}), for some μ, σ^{2}. We have

μ = E ∑_{j=1}^{d} θ_{j}X_{j} = ∑_{j=1}^{d} μ_{j}θ_{j}, σ^{2} = cov ∑_{j=1}^{d} θ_{j}X_{j} = ∑_{j=1}^{d} ∑_{k=1}^{d} θ_{j}θ_{k} cov(X_{j}, X_{k}).

□

Since the moment generating function of a Gaussian vector is the exponential of a quadratic form, there is no better way to express it other than using Linear Algebra. To this end, we think of the elements x of ℝ^{d} as column vectors. We use x′ to denote transposition, i.e. the corresponding row when x is a column. And, of course, (x′)′ = x. Consider the mean (column) vector

μ = (μ_{1}, …, μ_{d})′

and the symmetric covariance matrix

R = [r_{jk}].

Write also X for the column with entries X_{1}, …, X_{d}. We then have

Ee^{θ′X} = exp \{θ′μ + \frac{1}{2}θ′Rθ\} .

This uniquely defines the law of (X_{1}, …, X_{d}). This law is denoted by N (μ, R), where μ is the mean vector and R the covariance matrix.

Lemma 4.12. A Gaussian vector (X_{1}, …, X_{d}) is absolutely continuous if and only if R is invertible. In this case, its density is given by

f(x) = \frac{1}{\sqrt{(2π)^{d} det(R)}} exp (−\frac{1}{2}(x − μ)′R^{−1}(x − μ)) .

Proof Assume μ = 0, to ease notation. Assume R is invertible. It is easily seen that

R = E(XX′)

where XX′ is a d × d matrix and E(XX′) is the matrix formed by taking the expectations of the entries of XX′. This R has two important properties:

(i) it is symmetric (obviously); (ii) it is positive semi-definite, i.e. the quadratic form

u′Ru = ∑_{k} ∑_{ℓ} R_{kℓ}u_{k}u_{ℓ} ≥ 0

for all values of the variables, positive or negative. The reason for the latter is that u′Ru is the expectation of a non-negative quantity:

u′Ru = u′EXX′u = E(X′u)′(X′u) = E(X′u)^{2} ≥ 0.

Standard linear algebra shows that R has exactly d (counting multiplicities) non-negative eigenvalues. (In fact, they are strictly positive, due to the invertibility of R which is tantamount to det(R) ≠ 0). Furthermore, the eigenvectors can be chosen to be orthonormal. Letting U be the matrix whose columns are these d orthonormal eigenvectors and Λ the diagonal matrix with the eigenvalues in its diagonal, we have

RU = UΛ,

from the very definition of the eigenvectors. Now

U′ = U^{−1},

hence

R = UΛU′ = UΛ^{1/2}Λ^{1/2}U′ = P P′,

with

P = UΛ^{1/2}.

The matrix P is non-singular and is called the square root of R. We now define new random variables Z by

X = P Z .

The thing to observe is that the covariance matrix of Z is

cov(Z) = EZZ′ = EP^{−1}XX′P^{′−1} = P^{−1}R′P^{′−1} = P^{−1}P P′P^{′−1} = I,

I being the identity matrix. Thus

Ee^{θ′Z} = e^{θ′θ/2} = ∏_{j=1}^{d} e^{θ_{j}^{2}/2} = ∏_{j=1}^{d} Ee^{θ_{j}Z_{j}},

implying that the components of Z are independent standard normal. Hence the density g of Z is product:

g(z) = ∏_{j} \frac{1}{\sqrt{2π}} e^{−z_{j}^{2}/2}.

Now X = P Z, hence its density f is computed easily by

f(x) = g(P^{−1}x)/| det(P)|,

which yields the desired formula. □

If R has determinant zero then it is possible to “reduce the dimension” of the random vector X so that the density exists. In fact,

Lemma 4.13. The support of X is the range of its covariance matrix R.

Proof Suppose that R has rank r. Then it has d − r eigenvalues at zero, so that the matrix Λ consists of a d − r size block of zeros and the remaining non-zero eigenvalues. Hence now R = P P′, where P is a d × r matrix with rank r. We try again to find Z so that

X = P Z,

where Z is an r-dimensional random vector with density. If we manage to do this we will finish, since the range of P is the range of R. Observing that P′P is an r × r non-singular matrix, we pre-multiply by it to get P′P Z = P′X, hence Z = (P′P)^{−1}P′X. So if we define Z this way, we see that

EZZ′ = (P′P)^{−1}P′RP (P′P)^{−1} = (P′P)^{−1}P′P P′P (P′P)^{−1} = I,

i.e. Z is a collection of r independent standard normal variables. The formula for Z is the formula that solves a full-rank over-estimated linear system. We usually write Z = P^{+}X and call P^{+} the pseudo-inverse of P. It remains to show that every linear function of X is a linear function of Z. Let

F = λ′Xa , G = λ′P Z

with Z = P^{+}X. Consider

F − G = λ′(X − P Z).

Then

E(F − G)^{2} = Eλ′(X − P Z)(X′ − Z′P′)λ = Eλ′(XX′ − XZ′P′ − P ZX′ + P ZZ′P′)λ.

But

EZX′ = (P′P)^{−1}P′EXX′ = (P′P)^{−1}P′P P′ = P′.

So

E(F − G)^{2} = λ′(P P′ − P P′ − P P′ + P P′)λ = 0.
□

Terminology: If X has law N (0, R) with R having rank r then ∑_{j} X_{j}^{2} is called χ^{2} with r degrees of freedom.

4.8.3 Conditional Gaussian law

If it appears that we’ve done a lot of Linear Algebra, then this is because it is so: Dealing with Gaussian random variables (and processes!) is mostly dealing with Linear Algebra (or Linear Analysis!).

Without proof, we mention the following:

Lemma 4.14. Let (X; Y_{1}, …, Y_{d}) be a Gaussian random variable in ℝ^{1+d}. Then the conditional law P(X ∈ ·|Y_{1}, …, Y_{d}) is normal with mean E(X|Y_{1}, …, Y_{d}) and deterministic covariance matrix. Moreover, E(X|Y_{1}, …, Y_{d}) is a linear function of Y_{1}, …, Y_{d}).

EXERCISE 41. Let X, Y be independent standard Gaussian variables. Based on the last part of the lemma above, compute E(X + 2Y |3X − 4Y). (Hint: Make sure that (4.7) is satisfied.)

Probability

5: Important special distributions

Takis Konstantopoulos^{a}

                  www.smstc.ac.uk

\underline{Contents}

5.1 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5–1 5.2 Binomial, Poisson, and multinomial . . . . . . . . . . . . . . . . . . . 5–1 5.3 Thinning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5–3 5.4 Geometric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5–4 5.5 Uniform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5–5 5.6 Exponential . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5–6

5.1 Outline

We discuss several special random variables, and simple relationships between their distributions.

5.2 Binomial, Poisson, and multinomial

Consider the coin tossing experiment, i.e. consider a sequence ξ_{1}, ξ_{2}, … of i.i.d. random variables with

P(ξ_{1} = 1) = p, P(ξ_{1} = 0) = 1 − p.

The law of

S_{n} = ξ_{1} + ⋯ + ξ_{n}

is called Binomial with parameters n and p. From this we have

ES_{n} = np, var S_{n} = n var ξ_{1} = np(1 − p),

and, since ξ_{n+1} + ξ_{n+m} is Binomial with parameters m and p, and independent of S_{n}, we have that the sum of two independent Binomial random variables with the same p is again Binomial. We also have

P(S_{n} = k) = \binom{n}{k}p^{k}(1 − p)^{n−k}, k = 0, 1, …, n,

and, from the Binomial theorem,

Ee^{θS_{n}} = (pe^{θ} + 1 − p)^{n}.

EXERCISE 42. Compute the moment generating function of n^{−1/2}(S_{n} −ES_{n}) and show that, as n → ∞, it converges to the moment generating function of a Gaussian random variable.

EXERCISE 43. Use Chernoff’s inequality to estimate P(S_{n} > n(p + x)) for x > 0.

5–1

By letting p vary with n and taking limits we obtain a different fundamental law, the Poisson law. Specifically,

Lemma 5.1. If p_{n} = \frac{λ}{n} + o(1/n), as n → ∞, then

P(S_{n} = k) → \frac{λ^{k}}{k!} e^{−λ},

for all k = 0, 1, 2, . . ..

Proof Use Stirling’s formula. □

This is the Poisson law with parameter λ. The Poisson law is fundamental when, roughly speaking, we deal with independent rare events.

Let X be a Poisson random variable. Then We have

Ee^{θX} = ∑_{k=0}^{∞} \frac{(λe^{θ})^{k}}{k!} e^{−λ} = e^{λ(e^{θ}−1)}.

Differentiating a couple of times, we find

EX = λ, var X = λ.

Furthermore,

Lemma 5.2. If X_{1}, X_{2}, … are independent Poisson random variables with parameters λ_{1}, λ_{2}, … such that ∑ λ_{k} < ∞ then ∑_{k} X_{k} is Poisson with parameter ∑_{k} λ_{k}.

Proof We prove this for a finite number n of random variables (which is enough by the way we construct a probability on an infinite product). The moment generating function of X_{1} + … + X_{n} is

∏_{k=1}^{n} e^{λ_{k}(e^{θ}−1)} = e^{(e^{θ}−1) ∑_{k=1}^{n} λ_{k}}

and this is the moment generating function of a Poisson law with parameter ∑_{k} λ_{k}. □

Next we consider conditional probabilities:

Lemma 5.3. Suppose that X_{1}, X_{2} are independent Poisson with parameters λ_{1}, λ_{2}. Then, conditional on X_{1} + X_{2} = n, we have that X_{1} is Binomial with parameters n, λ_{1}/(λ_{1} + λ_{2}).

Proof Elementary conditioning: For 0 ≤ k ≤ n,

P(X_{1} = k|X_{1} + X_{2} = n) = \frac{P(X_{1} = k, X_{2} = n − k)}{P(X_{1} + X_{2} = n)}

= \frac{\frac{λ_{1}^{k}}{k!} e^{−λ_{1}} \frac{λ_{2}^{n−k}}{(n − k)!}e^{−λ_{2}}}{\frac{(λ_{1} + λ_{2})^{n}}{n!} e^{−λ_{1}−λ_{2}}}

= \binom{n}{k} (\frac{λ_{1}}{λ_{1} + λ_{2}} )^{k} (\frac{λ_{2}}{λ_{1} + λ_{2}} )^{n−k} .

□

Generalising this, we have:

Lemma 5.4. Suppose that X_{1}, X_{2}, …, X_{d} are independent Poisson with parameters λ_{1}, λ_{2}, …, λ_{d}, respectively. Then, conditionally on ∑_{k} X_{k} = n, the random vector (X_{1}, …, X_{d}) has law given by

P(X_{1} = n_{1}, …, X_{d} = n_{d} | ∑_{k} X_{k} = n) = \binom{n}{n_{1}, …, n_{d}} (\frac{λ_{1}}{λ} )^{n_{1}} ⋯ (\frac{λ_{d}}{λ} )^{n_{d}} ,

where (n_{1}, …, n_{d}) are nonnegative integers with sum equal to n, and λ = ∑_{k} λ_{k}, and where

\binom{n}{n_{1}, …, n_{d}} = \frac{n!}{n_{1}! ⋯ n_{d}!}

EXERCISE 44. Show this.

The symbol \binom{n}{n_{1}, …, n_{d}} is the multinomial coefficient since it appears in the algebraic

identity known as multinomial theorem:

(x_{1} + ⋯ + x_{d})^{n} = ∑ \binom{n}{n_{1}, …, n_{d}}x_{1}^{n_{1}} ⋯ x_{d}^{n_{d}}.

The sum is taken over all nonnegative integers (n_{1}, …, n_{d}), with sum equal to n.

The random variable (X_{1}, …, X_{d}) with values in ℤ_{+}^{d} is said to have a multinomial distribution with parameters d, n, p_{1}, …, p_{d} (where p_{1} + ⋯ + p_{d} = 1, so one of them is superfluous) if

P(X_{1} = n_{1}, …, X_{d} = n_{d}) = \binom{n}{n_{1}, …, n_{d}}p_{1}^{n_{1}} ⋯ p_{d}^{n_{d}},

where (n_{1}, …, n_{d}) are nonnegative integers with sum equal to n,

Of course, a multinomial distribution with parameters 2, n, p, 1 − p is a binomial distribution with parameters n, p.

5.3 Thinning

Suppose an urn contains n balls. There are d colours available. Let the colours be denoted by c_{1}, c_{2}, …, c_{d}. To each ball assign colour c_{i} with probability p_{i}, independently from ball to ball. Let S_{n}^{i} be the number of balls that have colour c_{i}, 1 ≤ i ≤ d. It is easy to see that (S_{n}^{1}, …, S_{n}^{d}) has a multinomial law:

P(S_{n}^{1} = k_{1}, …, S_{n}^{d} = k_{d}) = \binom{n}{k_{1}, …, k_{d}}p_{1}^{k_{1}} ⋯ p_{d}^{k_{d}},

where (k_{1}, …, k_{d}) are nonnegative integers summing up to n. Clearly,

S_{n}^{1} + ⋯ S_{n}^{d} = n,

so the random variable S_{n}^{1}, …, S_{n}^{d} cannot be independent. The question is:

Suppose that the number of balls is itself a random variable, independent of everything else. Is there a way to choose the law of this random variable so that the above variables are independent?

To put the problem in precise mathematical terms, let ξ_{1}, ξ_{2}, … be i.i.d. random colours, i.e. random variables taking values in {c_{1}, …, c_{d}} such that

P(ξ_{1} = c_{i}) = p_{i}, 1 ≤ i ≤ d.

Let

S_{n}^{i} := ∑_{k=1}^{n}1(ξ_{k} = c_{i}).

(In physical terms, S_{n}^{i} denotes precisely what we talked about earlier using a more flowery language.) Now, independent of the sequence ξ_{1}, ξ_{2}, …, let N be a random variable with values in ℤ_{+}. The problem is to find its law so that

S_{N}^{1}, …, S_{N}^{d} are independent random variables. (⋆)

It turns out that this is a characterising property of the Poisson law. We will contend ourselves by proving one direction:

Lemma 5.5. If N is Poisson then (⋆) holds. Moreover, if N has expectation λ, then S_{N}^{i} is also Poisson with expectation λp_{i}.

EXERCISE 45. Prove the last lemma.

5.4 Geometric

A random variable X with values in ℕ is geometric if it has the memoryless property:

For each k ∈ ℕ, the conditional distribution of X − k given {X > k} is the same as the distribution of X:

P(X − k = n|X > k) = P(X = n).

Think of X as a random time, e.g. the day on which a certain volcano will erupt. The property above says that if by day k the volcano has not erupted then the remaining time X − k has the same law as X, no matter how large k is.

Lemma 5.6. If q = P(X > 1) then

P(X > k) = q^{k}, k ∈ ℕ

and

P(X = k) = pq^{k−1},

where p = 1 − q.

Proof By the property of X,

P(X > k + n|X > k) = P(X > n),

for all k, n, which means that

P(X > k + 1) = P(X > k)P(X > 1).

Iterating this, we find

P(X > k) = P(X > 1)^{k}, k = 0, 1, ….
□

People refer to X as geometric with parameter p. The terminology is not standard because other people refer to X − 1 (which also has the memoryless property but takes values in ℤ_{+}) as geometric with parameter q. A matter of taste, really.

It is easy to see that Lemma 5.7. If X is geometric in ℕ with P(X = 1) = p then

EX = 1/p, var X = (1 − p)/p^{2}, Ee^{θX} = \frac{pe^{θ}}{1 − (1 − p)e^{θ}} .

EXERCISE 46. Do all that.

A concrete way to get a geometric random variable is by considering ξ_{1}, ξ_{2}, … to be i.i.d. with

P(ξ_{1} = 1) = 1 − P(ξ_{1} = 0) = p

and by letting

X = inf{k ≥ 1 : ξ_{k} = 1}.

We have P(X < ∞) = 1, so

X = min{k ≥ 1 : ξ_{k} = 1}

and

P(X > k) = P(ξ_{1} = ⋯ = ξ_{k} = 0) = (1 − p)^{k},

as required.

We have that

Lemma 5.8. If X_{1}, X_{2}, …, X_{d} are independent and geometric then X = min(X_{1}, …, X_{d}) is geometric.

Proof

P(X > k) = P(X_{1} > k, …, X_{d} > k)
= P(X_{1} > k) ⋯ P(X_{d} > k)
= q_{1}^{k} ⋯ q_{d}^{k} = (q_{1} ⋯ q_{d})^{k}.
□

EXERCISE 47. Let X, Y be independent and geometric. Show that

P(X − Y > n|X > Y) = P(X > n),

for all n, and interpret the formula.

5.5 Uniform

We have already seen, in detail, how to construct a uniform random variable, from first principles. Recall that U is uniform in the interval [0, 1] if P(U ≤ x) = x, 0 ≤ x ≤ 1. More generally,

X is uniform in [a, b] if, for all intervals I, the probability P(X ∈ I) is proportional to the length of I.

Of course, if X is uniform in [a, b], then cX + d is uniform in the interval with endpoints ca + d and cb + d.

Recall that if F is a distribution function and U is uniform in [0, 1] then F^{−1}(U) is a random variable with distribution function F.

Lemma 5.9. Let p_{1}, …, p_{d} be positive numbers adding up to 1. Split the interval [0, 1] into consecutive intervals I_{1}, …, I_{d} of lengths p_{1}, …, p_{d}, respectively. Let U_{1}, …, U_{n} be i.i.d. uniform in [0, 1]. Let

S_{n}^{i} = ∑_{k=1}^{n}1(U_{k} ∈ I_{i}), 1 ≤ i ≤ d.

Then (S_{n}^{1}, …, S_{n}^{d}) has a multinomial law. In particular, S_{n}^{i} is Binomial with parameters n, p_{i}.

EXERCISE 48. Show this last lemma.

EXERCISE 49. Let U_{1}, …, U_{d} be i.i.d. uniform in [0, 1]. Compute the probability P(U_{1} < U_{2} < ⋯ < U_{d}).

EXERCISE 50. Consider a stick of length 1 and break it into 3 pieces, by choosing the two break points at random. Find the probability that the 3 smaller sticks can be joined to form a triangle.

EXERCISE 51. Pick a random variable U_{1} uniform in [0, 1]. Let U_{2} be the midpoint of the interval [0, U_{1}] or of [U_{1}, 1], with equal probability. Continue in the same manner and define U_{3} to be the midpoint of [0, U_{2}] or of [U_{2}, 1], with equal probability. Show that the x ↦ lim_{n→∞} P(U_{n} ≤ x) is continuous but not absolutely so.

5.6 Exponential

A random variable T with values in ℝ_{+} is exponential if it has the memoryless property:

For all t, s > 0,
P(T − t > s|T > t) = P(T > s).

Lemma 5.10. If T is exponential then there is λ > 0 such that

P(T > t) = e^{−λt}, t ≥ 0.

Proof Implicit in the definition is that P(T > t) > 0 for all t. Hence α := P(T > 1) ∈ (0, 1). We have

P(T > t + s) = P(T > t)P(T > s)

for all t, s. Using this and induction, we have that, for all n ∈ ℕ,

P(T > nt) = P(T > t)^{n}.

This gives that, for all m ∈ ℕ,

P(T > 1 = m(1/m)) = P(T > 1/m)^{m}

and so P(T > 1/m) = P(T > 1)^{1/m}. Letting t = 1/m in the pre-last display, we have

P(T > n/m) = α^{n/m}.

Now, for t > 0, let q_{1} > q_{2} > … be rational numbers with inf{q_{1}, q_{2}, …} = t. Then

P(T > t) = P(∪_{k}{T > q_{k}}) = sup_{k} P(T > q_{k}) = sup_{k} α^{q_{k}} = α^{inf_{k} q_{k}} = α^{t}.

Since α < 1, we have λ := − log α > 0. □

We say that T is exponential with parameter (rate) λ.

It is easy to see that Lemma 5.11. If T is exponential with rate λ then T has density

f(t) = λe^{−λt}, t ≥ 0.

Also,

Ee^{θT} = \frac{λ}{λ − θ} ,

defined for all θ < λ, and

ET = 1/λ, var T = 1/λ^{2}.

Proof Note that

∫_{0}^{t} f(s)ds = 1 − e^{−λt},

showing that f is a density for T. The rest are trivial. □

Lemma 5.12. Let T_{1}, T_{2}, …, T_{d} be independent exponential random variables with parameters λ_{1}, λ_{2}, …, λ_{d}, respectively. Then min(T_{1}, …, T_{d}) is exponential with parameter λ_{1} + ⋯ + λ_{d}.

Proof

P(min(T_{1}, …, T_{d}) > t) = P(T_{1} > t) ⋯ P(T_{d} > t).
□

Whereas an exponential is the natural analogue of a geometric, in that they are both memoryless, the former also enjoys the important scaling property:

If T is exponential with rate λ then, for any c > 0, cT is exponential with rate λ/c,

and this is obvious.

EXERCISE 52. Let T_{1}, T_{2}, …, T_{d} be independent exponential random variables all with rate 1. Show that

law of max(T_{1}, …, T_{d}) = law of (T_{1} + \frac{T_{2}}{2} + ⋯ + \frac{T_{d}}{d} ) .

Another relation between geometric and exponential is the following: Let p be very small. Let X be geometric with parameter p. Consider a scaling of X by p, i.e. the random variable pX which takes values p, 2p, 3p, . . .. Then the law of pX converges to an exponential law with rate 1:

Lemma 5.13 (Rényi). For X geometric with parameter p,

lim_{p→0} P(pX > t) = e^{−t}, t > 0.

Proof P(pX > t) = P(X > t/p), and, since X is an integer,

P(X > t/p) = P(X ≥ ⌈t/p⌉) = (1 − p)^{⌈t/p⌉+1}.

Since lim_{n→∞}(1 + n^{−1})^{n} = e, we have that the last expression converges to e^{−t} as p → 0. □

References

[1] B. Fristedt & L. Gray A Modern Approach to Probability Theory, Birkhäuser, 1997.

[2] E. Hewitt & K. Stromberg Real and Abstract Analysis, Springer, 1965.

[3] A. N. Kolmogorov & S. V. Fomin, Introductory Real Analysis, Dover, 1975.

[4] D. Mumford, The dawning of the age of stochasticity.

   http://www.dam.brown.edu/people/mumford/Papers/Dawning.pdf, 1999.

[5] D. Williams, Probability with Martingales, Cambridge, 1991.

5–8